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FOREWORD 



'National Assessment of Educational Progress (NAEP), a project of the Education Commission of 
the Sitiies. is a research effort designed to gather information about the educational achievements of 
y-yeaNolds. 13-year-olds, 17-ycar-olds and adults (ages 26-35 J in 10 learning areas. Different learning 
areas are assessed every year and all areas are periodically reassessed in order to measure educational 
^han^e. Plans for assessment efforts, in the next (IVe years arp. outlined in the final chapter of this 
report. ♦ ' 

f BcQiuse of its focus on the collection and analysis of data, National Assessment must rely on profes- 
sional educators and organizations to study that data and drav^ out implications. and meaning. The 
project welcomes the serious interpretation c fits findings and realizes that only through such follgw- 
up work and dissemination can the -results have a bearing on education decisions and plans. We 
therefore welcome this report the result of a grant from NAEP to the National Council for the 
Social Studies (NCSS). The council organized for its task (see Chapter 1) and called on a number of 
its members to pariidpate National Assessment and NCSS are grateful to the many professional 
educators, listed on the following pages, who contributed to the study and thiS review. 



J. Stanley Ahmann 

Director 

National Assessment of Educational Progress 
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CHAPTER 1 



NATIONAL ASSESSFSiENT AND SOCIAL STUDIES 
EDUCATION: THE SETTING 

Jean Fair 
Wayne State University 



In the midst q{ unsettling social chmngc education 
becomes .inevitably a matter of publj^: debate and policy 
makini? The heat, even outright turmoil, of recent years has 
sub?5idcri, and faith in education as the road to salvation has 
t-iven wav. Still firm is the belief that education is basic to ' 
. mdividual and social welfare. And still wic^espread is the un- 
easv feelini; tljat schools are not doing what they ought to be 
domi,'. and not even doin^ well what they have long been do-^ 
in^ When too little money is at hand for public serv^es. tlie 
debate is further sharpened. o 

Fhe pressures in policy making make plain the need for 
information Simply stated, young^peepic,^ to school to 
■ learn something. Although the country hasSctrmulate^ in- 
lormation on scores of nO-doubt useful rpattersTTTttle-^ 
svstematic evidencp has been available on the crucial point 
iii'^ii alL actual educational attainment: 

Kfforts to ,t;ather evidence soon confront the basic ques- 
^"^'"^^f the debate: what should young people be learning, 
u l^^hould they achieve and to what extent are they doing 
so * I o these two ap* added others: how to find out and how 
to explain what is found in some way useful to decision mal^- 
int? Information gathering is no simple task: indeed, it is a 
subject of debate in itself. 

A major rffort ha^been under way by the National Asses- 
sment of Educational Progress (NAEP). Of special 
siE^ruficanre to social studies education are4ts recent reports . 
of findings in (Citizenship and social studies, 1969-70 and 
assessments./ 

Scime few ivords of background «re needed here. Formal 
discussion of the possibilities of a national' assessment of 
educational attainments began some 10 years ago. Sup- 
ported bv the Carnegie Corporation o{ New York, a private 
foandation, the Exploratory Committee on Assessing the 
Froi^ress of Education camfe into being in 1964 to develop a 
concrete plan. This committee's work became the basi^ of 
the present national project. In mid-1969 the Education 
Commi^ssion of the Staters (ECS), a compact presently of 
some 47 states and territories to consider and coordinate ef- 
forts and problems, assumed the governance qf the project. 

The prime purpose of the National Assessment of 
Educational Progress is to make information available to 
those mterested in education. Assessments are carricdon at 
retjular intervals, at the present in lOareas^ reading, writing,. 



science, mathematicif, social studies, citizenship, career and 
occupational development, literature, art and music ~~ in 
short, in much more tha)i' the three R's. NAEP assesses 
educational achievement, not merely school achievement. 
Obviously schools have responsibilities for education. Still 
television, magazines, libraries; newspapers, civic organiza- 
tions (especially those for young^peoplc), religious institu- 
tions, personal opportunities and experiences— these and 
pothers alf contribute to educafion. , National Assessment 
'does not aim to distinguish one. source of achievement from 
another. 

Moreover, the Assessment is not a national examination, 
a set of hurdles that students must pass over for continuing 
opportunity. Neither does National Assessment attempt to . 
^-oieasure the performance of any one person, school district 
or cvl?H.4tatc, nor award praise or blame to any institu- 
tion. Althotrgh^no^searLh can be made except from some 
fra me of reference, NAEP consistently refrains from in- 
terpretive explanations oTthe data collected. Even though 
perfection is never to be expecjcd nor debate cut off. 
National Assessment is a serious, highly prtjicssional under- 
taking. What can be learned from its efforts- 3(5serves attch- 
tion. 4 

Consequently, the National Council for the Social Studies 
.(NCSS) welcomed the opportunity for independent study, 
interpretation and dissemination of jhe assessments in 
citizenship and social studies, two sides'of the coin of social 
studies education. That NAEP supported this study with a 
grant of funds is a mark of its professionalism 

An^NfcSS Steering Committee took over all f espo i- 
sibility: Jean Fair, chaiiperson; June Gilliard; Dana 
Kurfman; James Shaver; and Ronald Smith. The task of 
dissemination has been the responsibility of the Steering 
Conrimittee. Five other tasks were identified, and major jn- 
vest'igators were appointed to give time and^thought to ex- 
amining them: (1) the assessment model. Bob Taylor; (2) 
methods and procedures, Guy Larkin; (3) the validity of the 
exercises, Francis^ Hunkins; (4) interpretation of the 
findings, Benjamin Cox; (5) consistency of the exercises 
with. NCSS Soc/fl/ Studies Curriculum Guidelines, desirability 
and realistically satisfactory performance levels, June 
Chapin. Each of the investigator*' reports was reviewird by 
the Steering Committee and, except for the first, by 
members of \.<t?k review panels. The first, a delineation ol the 
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assessment model, is accompanied by two commissioned 
papers speaking to" the model gind prepared by Joseph . 
Grannis and Michael Scriven. Those who worked on this 
project were not only comp^^nt but qf varying professional 
^ roles in social studies education; attention was given to rrfen- 
women ratitis, minority groups, and geographic areas. The 
project has aimed for thoughtful » honest and open points of 
view. ' ' 

This volume reports orf each o( the five ^asks of investiga- 
tion, but with emphasis upon what has been less fully 
'^presented elsewherd. Readers will find another emphasis in^ 
' a special issue of Social Education for May 1974, on Nation^ 
• Assessment in citizenship and social studies.^ The full NCSS 
report* to National Assessment is available through the 
Educational Resources Information Center (See inside front 
cover). Since hiary. people will find the publications of 
'NAEP itself usefi|], a bibliogfaphy is includefd in this report 
as Appendix P/^,e chapters in this ref^prt examine the as- 
sessments 'fr(Ht;i: -several stances. Both Taylor and Larkins 
treat, the, procr^ oC. arriving at objectives, methods and 
procedures, but in differing contexts. Larkins looks at the 
objectives and Chapin's panel at the exercises with NCSS 
Guidelines as criteria. Hunkins*s panel considered the validity 
of the exercises for the objectives, while Chapin*s panel con- 
sidered the v^orth of the exercises by the criteria of the NCSS 
Guidtlines. CcTx interprets findings from exercises categorized 
as social studies knowledge by the method of analysis, while 
Chaptn*s paneljudges realistically satisfactory performance 
levels for incfividiial exercises. 

The project believed National Assessment, as well as 
other assessment programs, is bettei" served by stating 
points of view that have bot}i xlifferencesand commonalities. 
The full NCSSVeport and this volume do attempt, however, 
J to consider basic questions: what is assessed, the extent to 
which the assessments can be counted on, what can be 
learned from tm procediijires themselves, what the findings 
mean and ti^e extent to ^which they ar& useful. 

So ambitious &nd pot*ientially influential an undertaking 
as a national assessment of educational progress must from 
the outset make decisions on a host of issues. Some are 
highlighted hei^ and elsewhere in. this report. 

A number of i;5sues cluster around the matter of objec- 
tives.' Who was decide, first^of all, what was to be assessed 
in citizei^tship^nd social studies — or any other areas? 
Teacher?? Administrators? Those in state departments of 
educaticjfn or those at the grass roots? Experts in social 
steadies Education? Scholars in relevant diciplines? Textbook 
authors and publisherst? Minority groups? Students? Those 
who support custom in education, *Vhat everybody knows 
it*s always been," or those who support innovation, "the 
^ cutting; edge?" Researchers on educational . problems? 
Educational policymakers in legislaturrs and school boards? 

Early In the enterprise and after consultation with several 
kinds of pe6pt% basic criteria for objectives were set. Objep- 
tlves had to be (1) considered important by scholars, (2) ac- 
cepted as\an educational task by the school and (3) con- 
sidered desirable by thoughtful lay citizens. "^ These criteria 
pointed to the kinds of people who were to decide, if not to 
the particulars of the 'process. 

Then contracting agencies, in a sense, experts in assess- 
ment, searched the literature of the field out of which came a 
tentative list of objectives. Panels of those competent in 
social studies education or the fields of social 
science/history, teachers and other school persons, and 
thougntiul lay people concerned about education reviewed 



and risvised until a set of objectives was formulated^. How 
many kinds of people should be heard frbm? Wepc a suf- 
ficient number of persons from minority groups included? 
Were there too many professional, comparatively well-off 
people, ancP too few who could see education froAi the van- 
tage of 'the poor? Were groups who needed information for 
policy making underrepresented? These are matters not 
readily resolved, especially within the necessary constraints 
. of time and cost, nor plosed off to future recQ/tsideration" At 
any rate, it is noteworthy tl\at tho»*ght from botli the profes- 
sional field and several sorts of people including laymen con- 
tributed to the formulation. 

Such a process does not make fora theoretically clean and 
consistent set of objectives. On the one hand, it is foolish to . 
disregard the contributions of scholarship. On th( other 
hand, ours is a changing society and a' pluralistic on.e. Not 
even scholars agree. No one official set of objectives exists, 
xior even one that draws wide allegiance, and probably least 
of all in the area of social studies education. Most of us- like it 
that way. It can be argued that no single satisfactory set of 
social studies objectives is either possible or desirable. 
Against the merits of a theoretically cor^sist^pt conception of 
social studies edi^cation must be balanced the need for a se^ 
of objectives that seem legitimate to many in society. 

Almoct from the start assessments in citizenship and 
social studies were separated. Much can be said iniavor of 
two assessments rather than one focUsed on an area of 
critical yet controversial importance. Many will support the 
idea that citizenship is the responsibility of ^e school as a 
whole, not merely programs in social ^studies, or e^n that 
citizenship is as much the responnsibility of out-of-school in- . | ' 
stitutions. But if social studies education can be thought of 
as emphasizing what is less likely to be learned informally in * 
the culture at large, it can hardly'be conceived as something 
without integral relatiori to individual lives and the require- 
ments of society. Nor^ can citizenship be defensibly con- 
ceived as social participation without thought or knowledge. 
Neither is it sensible to think of citizenship as primarily 
political and social studies nonpolitical. Issues of distinc- 
tion, overlap and emphasis are difficult- to resolve. 

Crucial also was the decision about what w^s to be assess- 
ed. National Assessment might have focused only upon 
some few basic skills, the three R's perhaps though they are , 
far easier to name than identify. NAEP might have focused 1 
its efforts on assessing knowledge and knowledge only in 
'some set of disciplines, or in learning areas commonly in 
school curricula. NAEP might haye attempted to assey the 
outcomes of typical, or presumably **bes{," or "poorest" 
school programs, or, for that matter, out-of school 
educational institutions. The list of possibilities is long. The 
actual decj[sion^wa£^for assessing a broad range of fields. As 
a consequence the assessment yields information about 
aspects of educational attainment in the population as a 
whole for which data have been sparse. Moreover, the deci- 
sion throws the weight of the Assessment to a broad rather 
than narrow conception of educational attainment, es- 
pecially important in social studies education. What' is as- 
sessed exerts powerful infiuence bn what schools see as im- 
portant to teach and what students see as important to 
learn. 

Another set of issues is <*fnbecjded in the closely related 
area of exercises, expected to furnish evidence of attainment 
of the objectives, to be sure, but also to be significant in 
themselves. When objectives are translated into exercises, 
the chips are down. 
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^ Citizenship and social studies are inevitably touchy* areas. 
To avoid what is controversial is in itself to take apposition. 
To the ^credit of NAEP, it chose not to rule out the con- 
.troversial. But how much and hp_w sensitiye? Review of exer- 
cises by panel? of several sorts of people recited in rewriting 
^or even dropping a substantial proportion of exercises.^ 
Social studies and citizenship were, indeed, more sensitive 
areas than some others. The issue here is much like that i;r 
^ the matter of objectives. What kind of balance can be ifad 
' bctvyccn the need for attending to the views of manv^iverse' 
social groups, and the necessity for devel6pifig^ exercises 
I legitimate jn the eyes of many in the country at large? 

more dilemma appears ins the matter of *'right 
answers, especially for thosc^ exercises about complex 
problems yielding to no simple solution and/or depending 
upon points of view and attitudes. Are hll positions, taken to 
be cohsidercd proper responses if their holders support them 
with whatever reasons? Or must some positions agree with 
predetermined proper responses, for example, sqpport for 
the rights of the first .AmenJmer?t? Or is some mixture ap- 
propriate? Blacks, native American Indians, to name two of 
a number of groups, experience significant differences in 
their social worlds from those of.dominant groups. What 
should be considered proper responses — or proper e/.or- 
cises — for such groups? 

Although National Assessment, unlike many school as- 
sessment programs, has not operate(j{?on a shoestring, it too 
must function within limits of financial- support, time and 
the capabilities of the general field of assessment. NAEP too 
must make choices to what extctit are self-reports in exer- ^ 
cises xustifiablc subsntuies for actual observation of 'Wive" 
behavior? How much effort should be devAed to developing 
•exercises iissessing more complex, highfcr cognitive and af- 
fective behaviors? And, indeed, if such assessment requires 
much time from respondents, hqvi^ much more time is feasi- 
ble without throwing the baby out with the bath? Could 
special substudies do the job? 

As does every assessment program, NAEF has had hard 
decisions in the construction of exercises. What they 
developed iy a far cry, but a heartening one, from what 
many'people have Jcarned to think of as "tests.'* Exercises i»; 
social studies and citizenship frequently utilize paper and 
pencil but also rdy on Interviews, and even obst^rvations of 
discussion tasks. If young people were asked to respond to 
the familiar multiple-choice forms, they were also asked to 
view pictures; listen to songs; use maps, graphs, cards from 
a library'carrd catalog and indexes; interact with each other 
in^ discussion groups; watch a film; and reply to interview 
questions. U not all, then many exercises are lively, in- 
novative, readily related to the present social world an^i ex- 
emplary. 

A last set of issues are those of interpreting the finding. 
Assessment in itself is neither evaluation, nor explanafion. 

As succeeding rounds of assessments in citizenship and 
social studies are carried on, benchmark data will be 
available. Such comparisons can be made now in science. 
"On most exercises measuring science knowledge and skills, 

achievement declined at all three school ages assessed 9, 

13, and 17 years" from the first to the second assessments,'' 
ri However, benchmark data can be had now for citizenship 
and social studies. 

As a guide to interpretation NAEP has developed 
national percentages of Successful performance for each ex- 
ercise (and for some groups in the population and categories 



of exepfcises). Although^ illuminating, a few illustrations 
make/It , plain that these^ performance levels are not neces- 
sarily standards of what is "good," "adequat<j" or even 
"bad." When asked in the citizenship assessment whether 

/a person on ttilevision or rs^dio should be allowed to state 
* any of three generally unpopular views, 'Statements that 
^4»ake some people angry," 3% at age 13, 17% at age 17 and 
24% of young adults would allow %11 three' statements and 
gave freecjpS^^ speech as the rcason.^ Somewhat highei* 
proportions would allow any one statenftnt.'' Wh'en askdd to 
identify the meaning of monopoly in the social studies as- 
sessment, 51% of the 17-yeai:-olds and 56% of the young 
adults could do so.'' When asked in the social studies assess- 
ment to read a line graph identifying retail prices for eggs 
^nd apples oyer a period of time, 89% of 13-year-olds, 96^^^f 
17-ycar-ol^'and 91% of young adults were able to do so.' 
But when the element of interpretation was included in an 
unreleased social studies exercise, fewer - 53% of those at 
age 13, 74% at age 17 and 69% of the young adults -'coqbd 
read and interpret a line graph. 

National Assessment also compiles results by groups: age, 
region, sex, color, parental education, and size and type of 
community. For each of these groups differences from the 
national percentages of success are available, and com- 
parisons among broad groups possible. Whether these dif- 
ferences are to be viewed with alarm or praise depends in 
part'upon the'size of the difference and the extent to which 
educational opportunity for all is accepted. What is more, a 

' the national percentage of success is judged too low, a more 
successful j^roup performance may still be inadequate. 

The proportion who are able "to do" an exercise depends 
in part upon the difficulty of the exercise. Many "standard- 
ized tests have been set up to distinguish the most able, 
typically able ai;id least able, and exercises constructed ac- 
cordingly. Although it might have been, such was not the 
purpose nf National Assessment. Instead, it aimed to 
describe the educational achievements of the population at 
four age levels. Consequently, NAEP developed exercises in 
three approx^ately equal groups: for the least able, 
typically able and most able. Successful performance 
percentages must be read and results interpreted accor- 
dingly. 

Of course, the Assessment might have followed still 
another path: hoping for mastery, claiming 'that everyone 
should be able to perform eVery exercise successfully. Such a 
course would have required either a set of exercises, within 
the reach of all, or a set of more-demanding exercises with a. 
built-in and higher "failure" ratfc. The former would have 
failed to tap what the more able could do.The latSer would 
have loaded the dice, emphasized not what has but what has 
not been attained. Much is to be said for "mastery." 
Perhaps we have all been too tolerant of "not getting it," 
moving or to something else before learning is achieved. 
The problem, however, comes down to agreeing on exactly 
what, specifically which, tasks every young person in this 
broad and' diverse land should be able to perform. 

The problems in, settling on proper difficulty levels arc 
again much like those in agreeing upon objectives. At any 
ratc'perfoffnance levels over, let us say 90%^ cannot simply ^ 
and in themselves be judged as satisfactory, nor those below 
as unsatisf^iory, pinpointing areas for improving 
educatiotial endeavor. 

National Assessment is alrViost inevitably caught between 
the frying pan and the fire. On the one hand, information 
from assessments can be better interpreted and used when 
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tied to a schoo! district or a sofeial studies prograpn. There, it 
seems at. least, influencing conditions might be sorted out. 
Better yet, research questions and accompanying and flexi- 
ble inquiry designs might point to influencing^ factors and 
implications for change. The meaning of data comes clearer 
whcji explanatory matters are related to outcomes, when 
questions posted give people a handle on the data. On ihe 
other hand, fears have been expressed from the outset that 
the Assessment might becomes a national testing program 
with all its consequent restraints. School districts or states 
or those, en gaged in some program or other — any iden- 
tifiable persons or groups could hardly be expected to en- 
joy or seek the glare of examination by some outside agency; 
??ometimes it is enough ip put up with those they can call 
their own. NAEP has intended to be neither a national 
testing program nor an examining agent. The thought of 
collecting data for ider.tifiable institutions, programs or in- 
dividuals on a national scale staggers the mind^ All thai 
means, however, tjiat National Assessment does not furnish 
information directly to those making policy decisions for 
particular; states or school districts or classrooms or social 
studies programs. 

Moreover, a national assessment was conceived at a time 
when federal ePbrts in education were growini^ and the 
spotlight was on educational attainment in the country as a 
whole. By now efforts have in many respects shifted towards 
states and localities. They have, in turn, their own needs for 
evidence that national assessment can satisfy only in- 
directly. Defensible assessment programs are costly, and 
shoddy or duplicate ones unjustifiable, all th'c more so in 
times of straightened economic circumstances. A mere col- 
lection of unrela"ted assessment dati from some states htre, 
,!>ome districts there could hardly allow for-coherrnce in the 
whole or for information gathered in one place but usi^ful in 
others. — — '~- 

In a sense, the strengths of N^ational Assesshroit — and 
many others at state »and local levels — have also been'Jts 
weakness. What factors are to explain the findings? What 
produces whrt? For which policy questions are assessment 
data to be provided ?NAEP is now addressing such* problems 
by undertaking '^special analysis activities requested by 
iJjjpE to answer questions pfU"tinent to federal policy 



decisions. . — for example, analysis of results for group 
combinations, such as race v.'ithin region and community, to 
provide information on the matter of whether **the federal 
government should devise efforts to redress resource im- 
balance and for whom?"*^ The project has also commis- 
sioned a study of background factors affecting school 
achievement with an eye toward the feasibility of including 
some of these in. .Assessment studies.** Perhaps there are 
other means by which NAEP can include consumers of as- 
sessment findings in some^vays like those of deciding upon 
objectives. 

Still some points seem clear and a few are cited here. 
Typical performance on exercises jrt otizenship and social' 
studies of school-age young people in the inner city is below 
the nation as a whole; typical performance of young people 
in well-off residential areas is above the riest of the nation.'" 
Obviously enough large proportions of inner city young pco- • 
ph are members of minority groups. Whatever can be said 
in support of arguments that the two assessments do not ac- 
count sufficiently for the experiences of-subcultural groups, 
it seems plaio that attention must b^paid. Social studies 
ei{Ication and t]ie multitude of conditions that infiuence it 
have to be better. The complexity of the problems does not 
justify sweeping them under the rug, 

.Social studies educators will do^well to look at the results 
' of specific exercises. Only ^\% of 17-year-olds can respond 
pr(;>perly to all five paFts of an exercise on using a sim{)le bal- 
lot. Nor do young adults pick this up once they become of 
voting age." Surely those in social studies education ought 
to take steps to see tftat 17-year-olds in their own schools do 
better on a matter so vital. A number of race-related exer- . 
cises show up a basic fund of decency . among young 
people.'- In the midst of cohflict and change social studies 
cduciition ought to find ways to. capitalize 'upon it Some 
^^JiZSuIoL Liyear-olds and 93% of 17-ycar-olds (and in- 
terestingly only 67% ^f young adults) believe that **teen-age 
' students should help decide whatccourses will be offered in 
their school system."'^ Such expifctatibfis-n,eed to be ac- 
counted for in SQcial studies curriculum planning. 

What has been done and what has been, found in the 
national assessments in citizenship and social studies is' 
worth thoughtful consideration. 
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CHAPTER 2 



POTENTIAL USES OF' THE NATIONAL ASSESSMENT I^ODEL 

AT THE STATE LEVEL 
AND FOR ACCOUNTABILITY PURPOSES 

• ' Bob L. Taylor ' 

University of Colorado. 



The basic task of this paper is to describe the model used 
by the National Assessment of Educational Progress 
(i\AEP) gathering data and reporting on the citizenship' 
area and to discuss the potential^uses of the model for state 
assessment, curriculum developn»cnt and accountability 
purposes. It is not \yithin the scope of this paper to criticize, 
the model with respe9t to technical flaws; hence, the model 
is descHbcd and discussed j^ithout reference ta problems of 
'^design. 



The Model 



The model is in the continuous process of being refined 
and improved; thus only its basic components are presented 
in Exhibit 1. A circular scheme is used in presenting it since, . 
in reality, its actual application may be initiated with any 
one of the components. Also, in its actual applicationi there 
are continual interactions between and among the Various 
components. While theoretically the process starts with the 
refinement of overall national go^Is into specific subject- 
matter, behavioral objectives and progresses in logical se- 
quence through to the final utilization of information,- in 
practice there is much greater freedom with respect to the 
• utilization of the components. 

The model for the citizenship assessment is presented 
here in outline form with a fairly detailcid description of its 
components. As presentedin Exhibit 1, there are seven basic 
components identified in thcjr^odel: objectives development, 
exercise develojament, sampling plan, administration of ex- 
ercises« scoring and analysis, reporting and dissemination 
and utilization of information. M hile many of the fine points 
of the model are not developed in the following outline, it is 
described in sufficient detail to give the reader a good under- 
standing of how the data were collected and what implica- 
tions might result from these data. The number of subtopics 
In the model and their distribution indicate that the major 
efforts of National Assessment. have been with the first five 
components. The last three^components have been areas' of 
controversy; therefore, they have received less attention un- 
til recently. ^ • 



Outline of the Assessment Model for Citizenship 



Objectives Development^ ^ ^ ^ ^ 

A. The tasH of developing objectives in the field of 
citizenship Was awarded to the American In- 
stitutes for Research (AIR) of Palo Alto, 
Californi|i. -These criteria wctt used in examin- 
ing the objectives: ' , 

1. They were considered important by scholars. 

2. They were accepted as an educational task 

by the school. . 

3. They were considered desirable by thought- 

ful lay citizens. 
Scholars' reviewe'd th^ objecttves for authenticity 
with respect to their subject fields, school people 
.reviewed the objectives in terms of their actuail 
emphasis in their schools, and laymen reviewed 
them in terms ot their value in life. 

B. The AIR staff reviewed previous lists of 
citizenship objectives and reduced these to one 
compreticnsive list of 20 objectives. 

C. Outstanding local teachers familiar with each 
target age level (9, 13, 17, adiftt), working 'With 
the AIR staff, broke down each general objective 
into the most germane behaviors dieemed ap- 
propi iate as goals for a given age level' 

D. A selected group of students and adults in each 
age level was asked by the AIR staff to recall 

^and describe outstanding citizens of their ac- 
quaintance . and specific incidents reflecting 
good and poor citiz^enship. Th<tse incidents and 
descriptions, about 1,000, we||e used to check<^ 
the completeness of the initi^^l Jist of objectives. 

E. The objectives ♦ were stated jon three ievcj^: 
general objectives, subobjectives and behavioral 
age illustrations or statements. The results were 
summarized for each age level 

F. ^The r^evised list of objectives, broken down intd 

important behaviors, was then studied for three 
days by a panel of national leaders in citizenship 
education and related social sciences. 
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EXHIBIT 1. Compohcnts of the Model 



^ NATIONyVL GOALS ^ 
Objectives Development \ 



Exercise ' 
. Development 



Scoring . 
and Analysis 



^Administration 
of Exercises 

. 1 



Sampling 
Plan 
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G. A gtoup of persons in various roles from selected^ * 
Qalifornia communities reviewed the objectives 
and made sug^stion$. These persons included 
public and pr^ate school administrators, 
counselors, teachers, a jiklge, a county planner,-^ 
labor and business leaders and sqciaL scientists. 

H. The objectives were then reviewed by panels of 
•laynien. Elevea lay'j^vi^w^ panels representing 
four geographic areas of the»country and three 
different* cqmmunity sizes we^c used, kach 

p. panel spent two days reviewing the objectives 
^ based on^^these two questions: *^*Is~ this 
* something important for people^to learn today?** 
and '*Is this something I wdu1d*like'tp have my 
^ children learn^** 

Exercise Development^ ^ * » » 

A. The production of the edtercises was initiated by 
4IR in 19.66. The exercises were developed to 
cover all of the major objectives and to represent ^ 
the selected content areas. Many exercises re- 
quired the use of interview techniques, as* Well as 
the usual peQcil-and-pape/exevcises. Self-report 
and group-task exercises were also used. 

B. Because NAEP intends to describe what people 
,in an age level can do, the exercises wc^re writte 1 
to reflect three difliculty levels — jfnowledge or 

i « skilts common tb almoj^t all persons in an age 
level, skills or understandings of a Atypical 
meni^er of an age level, and understandings or 
knowlecjge developed by the most ab%^ persons • 
in an age- level. 

C. AU exercises' were developed to n^eet these 
criteria: content validity, clarity, functional ex- 
ercise formatj'^clusterin^ extrcises based on a 

.. single set of stimulus materials, directionality of 
response, difficulty level, pontent sampling and 
' .overlap between age levels, The exercises were 
• ^ direct measures of some pieces of knowledge, 
understandings, attitudes or skills that were 
mentio^ied in one or more of the objectives. 
The exercises were 'reviewed by panels of lay^ 
persons for clarity, mean ingfulh<!ss arid invasion 
of privacy. 

There was a tryout of tHe exercises involving 
representatives of groups in the actual assess- 
ment — regions, communities^ races, sexes and 
aK^vefs. Following the tryouts, the AIR staff 
ainK]lubject-matter specialists reviewed the' 
^tryou^data and made needed revisions. 

F. A conimiitee of subject-matter specialists, 
measurement specialists and NAEP staff 
members ^t«^he exercises to be included in- 
the packages according to a set of criteria; the 
exercises were selected based on the ratings. 

G. The selected exercises were reviewed by U.S. 
Office of. Education personnel for any infringe- 
ment of privacy on the'part of the respondents or 
possible offensiveness. 

H. Since there were about 160 minutes of testing 
time available for each age level in each Learning 
area, the exercises used were only a smaii 'jam* 

^ pie of the potential number of exercises. The e.'^- 
ercises were assembled into administrative uhits 
(packages) for groups up to 12 persons. 
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III. Sampling Plan* »o 

A. The sampling |)lan was subcontracted to ' 
Research Triangle Institute, Raleigh, North 
Carolina. A muftistage design, 1 which was 
stratified by region, size of community and \ 
socioeconomic status, was used.^ This 
probability sample allowed researchers to col« 
lect data from a small sample ofHhe population 
and to infer Trom thai sample -certain 

^ characteristics of the^entire population. 

B. The populations for assessment were all 9«>year* 
olds, all IS-year-olds"^, all l7-year*ohds and ^all. 
young adults 26 through 35 years old in the 50^ 
states pluk the District of Columbia. The only 
exceptions wdre^ the exclusions of in- 

^ stitutidnalized pdividuals of these ages (those in 
^ hospitals, prisons) and ^itbi^rs who could not be | 
reached. \ ^ ' 

C. Fqr'ages 9 ana 13, only a school sample was 
used, and for^the 26*throu§h^5 age level only 
household sample was.Ujsed. For the 17-year- 
olds, both a schoq) and a household sample were 
used. ' 

D. The entire country was divided into population 
areas as follows ii^^cities, counties exclusive of 
cities and pseirdbcounlies — two or more coun- 
ties were put together when the population of a 
single countyA^as less than 16,000. Each pop« 
ulation unit of 16,000 residents, was assigned a 

y o number. ^ 

E. ^ The country also was divided ' into four 

geographic regions: Northeast, Southeast, 
' Central and West. ' 

F. Each geographic region was divided into com- 
mu)H|ties of four types: large cities of above ' 
l80,0w population; urban, fringe, nr)(ddlt-si2ed 
citiey between 25,000 to 180,0P0j)opulation and 

^ small town/rural areas of under ^25, 000 popula- 
tion. 

G. The 52 sampling units for each geographic area 
c ' were spread across the four community types in 

^ a fashion proportional to* their population in 
relation to the at«a population. 

H. To insure comparable represeiitation from each 
part of .the couipy, an equal number of sampling 
units was selected from each^geograpliic region 
— 32 from each of thC'four regions for a total of 
208. 

I. Sampling units Were selected at random. This 
plan di^ not guarantee that ali 50 states would 
be included in the sample. This was not a survey 
objectiye, but later the design was changed so 
each state was included in the sample. 

J. In "each sanipling unit selected, all school 
bujldings enrdlling students of the sample ages 
(public, private and parochial) were idehtjiried. 

K. The plan for Schools was to select units <>{ ap;^ 
proximately 250 to 350 pupils for each age level 
and from at least two different buildings within 
e^ch sampling unit for each age level. 

L. Each cooperating building principal provided a 
list of names of students in the building from the 
^ specific age levels, lliis list was used for the final 
random selection oPstudents to take the assess,- 
ment' exercises from that building. | 
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M. Information abouf the areas was obtained from 
U.S. Census data. In order tS insure reliable in- 
formation for lower-socioeconomic groups, these 
groups wer^ oversamplcd. There was a* dis- 
' pr^ortionate number of schools from \ lower- 
socioeconomic areas included. In the overall 
results, the data from the lower-socioeconomic • 
areas were given the percentage v^lue in which 
they occurred in the total population. 

N. From each of the 208 geographical samples, 100 
adults^ ages 26 through 35, were randotnly 

^ selected using the following procedures. Each of 
the 208 'geographic samples was divided into 
equal secondary sampling units. Then 10 secon- 
dary sanppling units were randomly selected 
from the total 208 samples. Interviewers then 
personally contacted the people in the chdscn - 
secondary sampling units of the 26-xhrough-35 
age level and out-of-schoQL-4?-ycar-olds^ These 
persons were asked to participate in the assess- 
ment. 

(). Individuals were classjficcl as black, v^hite and 
other on the basis of linformation "provided by 
the school or by pbservation. Results were given 
for black and v^iiite only. The number of in- 
dividuals classified as other *was too small to 
produce j:ci«Tfeie results. 

t .Administration of Exercises" 

A. .Administration of the exercises was sub- 
contracted to Research Triangle Institute in the^ 
East and to Measurement Research Center of 
Westinghouse Learning Corporation, Iowa 
City, Iowa, in the West. Cooperation of schgols 
was obtained by first contacting officials at the 
state and ^ then at school district levels. There 
was above 90 percent cooperation^ by schools. 
.A'dults and out-Of-school 17-yc<»r cjds were con- ;, 
tac.ted by a personal dQ^l;^>o-door household 
canvass.. Each out-of-scmool participant was 
contacted individually. The right of each to' 
refuse to cooperate was respected. 

B. A full-time trained staff of 27 district supervisors, 
' managed the Tieldwork. They v^ere assigned to 

different geographical areas of the United , 
Stafes. They contacited schools and recruited 
and train'ed local teachers to help in the ad- 
ministration of the exercises in schools; they 
recruited and trained other available persons fqjf- 
^ the out-of-schoohadministration. 

C. In the schools, students from ajsingle age level 
' from differcnfc'Qlasses were brought together in a 

room for excraise administration. Group size 
was. at least 8, arid usually 12, studen'.?. 
' I). The exercises were organized ic* ^packages that 
contained exercises from two or three different 
learning areas at a single age level/ No one 
person took all the exercises in his age level. Age 
levels" were 'assessed at different times of the 
year. 

E. In packages administered to groups, taped 
^rcttions and taprJ readings of the exercises 
were used il?addition to printed packages. This 
was done to establish consistency in timing and - 
administration plus to provide for nonreaders. 




F. Several jpaekages at ages 9, 13 and 17 consisted . 
; of exercises that were giverr by eJcercise ad- 
ministrators to on'e "individual at a time. The ad- 
ministration of all tne packages for the ^dult as- 
sessment was done by interviews. 

G. Each paj^gc required about 50 minutes of ad- 
• minis^fative time. Each person took only one 

package with the exception of the out-ot-school 
17-year-oldsi*wuo were asked to take four or five 
packages each since they were^^h^; most difficult 
and expensive group to locate. 

H. Students' nam<?s were confidential and did not 
appear on any packages. The name roster was 
kept at tjhcjiuilding level and used only in the 
oFgan^ation of the in-school sampling. 

v. Scoring and An^lysis^* *^ ^ ' 

.\. The scoring and analysis of the exercises were - 
subcontracted to Measurement Research 
Center of Westinghouse Learning Corporation, 
Iowa City. Iowa. 
.B. The multiple-choice exerciser wei;e scored and 

recorded routinely by machine. ^ 
C. The open-ended exercises were scored by 
trained professionals using a key of acceptable 
and unacceptable achie,vements in^terms of tne 
* objectives. 

f). Results v^ere i"cported for each goal. Al^o, the 
results were reported both as (1) the percentage 
of any group of respondents making the desirda 
responses, to an exercise and (2) the difference 
between the percentage of a group making the 
desired responses and the xsor responding 
national percentage. 

E. In the assessment, there was a'Wck of phopor- 
tionality among characteristics used in the com- 
parison of groups, such, as colof, sex, parental 
education. AySt^iifiiical procedure, balancing, 
was used to correct for this problem in the com- 
parative analysis of the data. Balancing is a 
procedure to examine the performance of groups 
clas^fied on one characteristic adjusting for the 
fact rhsit these groups differ on a specified set of- 
other characteristics. 

VI. Reporting and Dissemination'^ 

A. The repor|iog of results was directed to subject- 
^ matter specialists, professional educators and 
informed laymen. Multiple reports were 
J developed to serve these different audiences. 

B Approximately 40 percent of the exercises were 
reported at the ertd of each assessment year. Not 
all exercise were reported since they were to be 
used over again in future assessments'^ in order to 
measure change. 
*C. The exercises released Tor publication^ were 
selected to be representative of all exercises ad- 
/ ministerH as well as the results received on the 

ass'^'iiment. "* • / ^ 

I). Reporting was done by age ' levels. Since 
some exercises were used with different gge 
' levels, there were often comparable data across 
two or more age levels, 
E. Reporting vs^as also done by groups within the 
categories of regions, community types, sex, 
* socioeconomic status and cdlor. 
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F. Rcpohs were printed with a short description oC 
the exercises, the national percentage of success 

^ and gri^p differences* from the national perceniv^ 
tage ofiiccess for each exercise and withojit in« 
tcrpretation of results. ^ 

G. Both observed and balanced results for all exer- 
cises were reported by groups. The effects of 
balancing on measui^ed characteristfcs such as 
sex and region were included in the report. 

H. There were tio scores reported for individuals. 
No single indivi$lual took more than one twelfth 

> ^of they' exercises, and no individual tooj« a 
. - ' package that ^aniplcd a single learning area. 

I. Results were reported through the medi^f|fvnt- 
ten word, radio, television, films and personal 
%o reports. * ' 

Vli. Utilization of Information^'* *^ 

. A. The results provided potential information for 
education decision making. For example, ^con- 
sidering the somewhat lower performance ofjthe 
Southeast region on the citizenship re^iflts, 
school boards in that region might decide to put 
greater stress in their school programs on 
citizenship skills, ^ndcrstandiijtgs and at- 
" • titudes.'" o 

. B. . The ijcsults raised many questions that may lead 
tip other investigations. For example, m making 
comparisons of al! citizenship results combined, 
it was found that the extreme affluent suburbs 
show,ed substantial median '.advantages at, all 

(ages and that the extreme rural and extreme in- 
ner city showed substantial deficits at all ages. 
. ^Chese discrepancies in performance need causal 
sidfdies conducted from thp perspeictives of dif- 
ferent disciplines such as political science, 
sociology, economics and education.-' Com- 
parisons might lead to other studies: a sample of 
, 5-year-olds might be asseiged for a basis of com- 

parison; results might oe broken down by 
states; or new learning areas might be in- 
vestigated, y ^ 
C. The results o(\ several cycles should- provide 
evidence of the chang? in knowledge, skills, un- 
derstandings and attitudes in the age levels as 
' they relate to education objectives. 
O. School administrators can make comparisons 
be^een groups and may improve student per- 
formance- from the information gained in this 
. manner. , ^ 

From this review of the model, it is evident that the NAEP 
staff has put a great deal of effort and know-how into the 
design, plus the development, of each of the components. 

In summary, it can be^ointed out that the model reflects 
some^important choices on the part of the NAEP staff. The 
^ decision was made to assess a broad range of objectives in 
each of the 10 learning areas. Certainly, it would have been 
' much easier and cheaper to have concentrated on a narrow 
set of objectives. .Also, the effort was successfully made to in- 
clude the higher cognitive levels in the assessment exercises 
and to deal with the affective domain. In learning areas like 
citizenship and social studies, the usual standardized test 
concentrates on factual knowledge, which is of a less con- 
troversial nature and easier to assess. National Assessment 
should be commended for Jt.s bolder, more comprehensive 



^pproaoi^ to the lask, which* searches fV more significant 
^' jkids of data. 

^ . Use Of the Model at State Level 

National Assessment a^census-like smdy to collect in- 
» formation concerning> the ediicationaP attainments of 
Americans. iTn pknning for the collection of this census-like 
data, the model, which was presented in the previous sec- 
tion, was developed. A number of states have found adapta- 
tions of the model useful in conducting state assessments in 
which desirable learning outcomes arc identified and the 
• status of learners with resprct to these outcomes is deter- 
mined. 

State assessment is a rapidly developing movenrient. At 
this writing/ ail of the states have assessment activities either 
•.iij operation, in a, developmental proc<^s or in a* planning 
stage.-- While iM statewide assessrient programs have 
^ many similarities, they break down into two Basic types of 
programs on tlie questioki, **Who gets to u§c the results?" 
The>^ivisions are those states for which d^ are collected 
for decision making by state agencies and those states for 
which data are collected for decision making by teachers 
and administrators. Stale programs for' which the emphasis 
is on collecting- information for state-level decision making 
are: Arizona, California, Colorado, Contiecticutj District of 
Columbia, Florida, Maih^, ^Massachusetts, Michigan., 
p Nevada, New Jersey, Ncw,Vork7^ftiorth Carolina, RhoSe , 
Island, South Carolina, Tennessee and Texas. Programs for 
which the emphasis is on collecting information for* local- 
level decision niaking are: Alabama, Arkansas, Delaware, 
Hawai^ Idaho, Iowa, Kentucky, I^Iississippi, New 
Hampshire, Ntw York, North Dakota, New Mexico and 
Pennsylvania.-^ . 
^ • Beers and Campbell report that a number of 
characteristics are appearing in these assessme^it p^ografhs. 
In about a'^ third of the states, the programs wpre'^mandated 
by the state legislatures, •and the r^esults of the assessments 
are to be reported ba<"k to the state legislatures. In a few of 
the states, the data are to be used for Planning, Programm- . 
ing^ and Budgeting Systems (PPBS). In about half of the 
states where tj;ie assessment data are being used to make 
state-level decisions, state and federal funds will be allocated 
based on the results. Participation in assessment^ is required 
by law in about a fifth of the states. In many states wh^re the 
assessment datal^re being used to make state-level deci- 
sions, samples rathcithan all students are being assessed, 
while at t^he local level, all studetits-in the target populations 
are being assessed. Critericfn-referenced instruments are 
very common with the states where the data are being used 
for state-level decisions, bur the states collecting information 
for local decision making are favoring norm-refcftcnced in- 
struments. Finally, no dominant funding pattern has 
evolved in either of the two groups of states.'* 

State Adaptations of the Model 

In its assessment of citizenship education, Maine madt an 
extensive application of the NAEP model and carefully 
duplicated it to collect comparable data at the state level. 
Maine's first cycle of the 10 'earning areas of the Assessment 
is to be completed by scheduling two of these areas each 
year for five years. Citizenship and writing were the first 
learning areas to be assessed. 

Based on the results of a previous -study of objectives /or 
education in Maine, two review committees decided to ac- 
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ccpt* the National ^As^essment objectives as being closely 
^ related ra'the Maine objcctives.Maine selected the 17-;year- 
' old population of in-school students for its first assessment. 

A sample of 2,000 17-ycar-old students was used tp repre- 
sent the approximately 17,000 17^ycJir-old students in the 
state. The state was divided into four geographical regions. 
As in Na/tional Assessment, schoollbuildings were randomly 
- selected from 4 he geographic regions, arid students were 
I .then rahdorfkly selected from the buildings. .Packages were 
J developed with -exercises taken from the two learning a^eas. 
The available, released exercises irom NAEP^were carefully 
examined to see if they reflected objectives valid for Maine 
and to Sec if some could be modified, where needed, to be 
administered in grodp sessions using the paced-tape method 
while slifl detaining a high degree of comparability to ti^ 
National Assessment individually administered exercises. 
JThe packages were made up of 23 citizenship and 7 writing 
^ exercises, plus a 23-item student questionnaire. T^e exercise 
format was kept virtually' identical to the one used in 
National- Assess 5Tfent. Trained administrators were sent out 
to administer the exercises, and the exercises were scored 
according; to I^AEP procedures. On data reporting ^nd 
analysis, there was the census-like reporting of the perfor- 
rtiance of the Maine studxintH>plus comparisons of the Maine 
results with appropriate National Assessment data. 
' In nummary, the Maine assessment duplicated NAEP 
^ procedure as nearly as possible. With minor exceptions, the 
same objectives were used for citizenship. Th^jSamc sampl- 
ing design was used with adaptations smaller 
geographic area and population. The exercises* were, for the 
most part, taken from those released by National Assess- 
ment, and they were organized into packages similar to 
those used by Nati6nal Assessment. The administration and 
scoring of the exercises were conducted in the same manner 
as NAEP. Since M^ine used jhe same private contractors as 
National Assessment, the duplication was complete 
wherever possible. The reporting and data analysis were 
similar, and the data did provide the opportunity to com- 
pare Maine's results with .those of ^^§^"^1 Assessment. 

Here, the model was very carefiJ|^^^i^icated at the state 
level. The big question that c^es to mind after studying 
the Maine citizenship report is, "Aren't the National Asses- 
sment data being treated here as spme kind of a national 
nAtrq against which the performances of 17-yearrold,Jiu- 
d^ts in Maine were being compared?" Of course, ihis use 
of Assessment data had bJen questioned from the start of 
the proposal for an assessment at the national level. Now 
Maine has provided the opportunity to study the effects of 
this use of the data on the cducatidn system of a state. 

Another state that carefully followed the model was Con- 
nccticut,2f> where an assessment was Jlrst conducted in 
reading. To permit comparisons, the Connecticut program 
used available ins|ruments and applicable procedures 
developed by NAEP, but adapted to the requirements of the 
local situation. Connecticut's reading objectives were 
matched to the reacling objectives of National Assessment. 
Approximately 220 reading exercises from NAEP were used 
in producing t e packages used in the Connecticut assess- 
ments Exercist<( w^cre selected to represent all 0/ Connec- 
, ticui*s reading objectives. The age levels assessed were 9, 13 
and 17. As with the National Assessment packages, tape- 
recorded instructions wer'^used. The sampling design was a 
multistaged one duplicating with few exceptions the 
National Assessment design. As with National Assessment, 



a group of administrators for the packages was fctcruited 
and^trained. 

, The Connecticut assessment was another example of 
careful duplication of the NAEP raoder— even usin^the 
same objectives and exercisesi Again,^National Assessment 
X ;sults*were used as norms to which the state results were 
compared. 

Th^^Texas Needs Assessment used the model for the 
development of their assessment in mathematics at the 
sixth-grade level. However, while usin^ ideas from the 
model, the Texas people broke with it in a number of places, 
\^ They were concerned that tKc assessment would yield infor- 
mation that y^outd be useful to teachers in their classroom 
instruction ofstudents. From a pilot study, it^was decided to 
use a criterion-refel^nced reading test and to work with 
#grade levefs instead of age levels ofstudents. They worked 
J with the sixth grade, and the tests were administered by the 
staff of each school t,hat participjrted in the assessmt.nt. The 
objectives were chosen from the major skill areas treated in 
the state-adopted textbjfOtcs. Regional location and com* 
^ munity size were taken jn^ consideration in selecting the 
sample. Approximately 1 (Percent of the Texas schools 
teaching at the sixth-gfade level administered tests, and ap- 
proximately 10 percent of the pupils in the sixth grade were 
included in the sample. Reports were gi/en to teachers on 
the performance of their individual students. Also, there was 
a school report on the performance o^he students for eacli 
school and a report on each of the clashes in the school. 
Comparisons were ^ade on the basis of sex, race and size oL 
community. ^ / 

The Colorado Needs Assessment, while using the model, 
made an even greater break with it.^" Its objectives were 
ba^ed on a state study of education goals, and these goals 
were restated 4n terms of perfqrmance objectives. Following 
the model,- objective-referenced exercises were written. A 
sampling design was used and the student responses were 
analyze^. In this assessment, classroom teachers wer^ in- 
volved in the writing and refinement of the beh£ivioral objec- 
tives. Qljiiective-f'eferented exercises werg written for nine 
learniy^ areas. The exercises were administered to a sample 
of 30,pOO Colorado students. A stratified, random-sampling 
procedure was us^d to ^elect a sample of school districts of 
. ihe state. Then schools were selected at random from the 
'^districts chosen. Finally^ classes in school buildings were 
^ randomly chosen for testing. The samples were represen- 
tative of all Colorado students \n grades 3, 6, 9 and 12. A 
group of proctors was hired and trained to administer the 
exercises, and the exercises were scored by computer. The 
data were analyzed on a statewide and district basis, and 
the results were broken* down by groups, e.g., boys, girls, 
urban, rural. 

As pointed out earlier, ^tes are rapidly moving into the 
assessment field. Somc^re reproducing the NAEP model at 
the state level, and others are developing variations of the 
model. The cruder efforts Have resulted in endless pages of 
raw percentages without any explanation of the results^ 
^ Based ona/stirvey of state assessment programs, Beers and 
CampbefTidentified several of the problems that are com- 
mon to these state programs. ^'^ Naturally, a shortage of 
money and staff was the most frequently mentioned 
problem, for it»is a fact that many states have moved into 
this area without providing adequate funds ^or a realistic as- 
sessment program. Also, teacher resistance to assessment 
and negative public attitude toward outside testing vyere 
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problems mentioned. Test reaults have been misused in the 
^ast; teachers have been fired, for exapiple. on the basis of 
incorr/ect interpretation of test results)i»Also, test developers 
have, been guilty of vio\ating the privacy of studen*slhrough 
questions that transgressed the examinee's human and Ipgal . 
rights. A third problem area has been with the utilization 
and dissemination of resaults. Some school officials do not 
understand the results. In some situations, thcrV has been 
hostility to the resultr. Some officials have ignored results tn^ 
making decisions. Finally; results have frequently mot 
reached the right people in a usable form. ^ * * 

. Adaptation of the Model 

In the above discussion, it is evident that there will be as 
niany adapltSrions of the mo^el as there are state units con- 
ducting assessments. Probably there is no spofcific assess- 
ment model that^is the best; hence, there is no model that 
should be applied without modification in any and all situa- 
tions. Nevertheless, there are principles of good assessotent 
. that should be applied in developing or adapting a model for 
state assessment purposes. Listed below are some 
characteristics that should be found rn a good assessment 
program.*" \ 

1 . The program has clearly defined goals that apply to a 
particular audience or audiences. 

2. The program has a realistic nupnber of goals that are 
attainable under the existing assessing conditions. 

3. The program has established priorities among its 
goals and places its major efforts on its major goals. 

, 4. The program has been designed to gather informa- 
tion considered to be important in education. 

5. The program has specific objectives, which it is striv- 
ing to attain. 

6. The program has been designed to provide results at 
a usable level o/ accuracy. . • ^ 

7. The program has used data-gathering instruments 
that measure tl^e objectives of the assessment. 

8. The program has collected data in such a manner as 
to introduce a minimum of error in the results. 

9. The program has scored and proceed data in an ac- 
curate manner, >^ 

10. The program has used anal)6tTC techniques that 
provide the data breakdowns needed by decisionmakers. 

1 1 . The program has reported results in a manner usable 
by its audience. ^ 

12. The program has provided help in the interpretation 
of results and assistance in their implementation. 

13. The program has provided for the active involvement 
of groups of persons from all of the major audiences for the 
assessment results. 

Implications of the Model 
for Curriculurn Development 

Of course, a major, potential outcome of Nat\onal Assess- 
ment and the model was providing ne>v, accurate data with 
regard to curriculum problems. Curriculum decisionmakers 
are furnished data that have not been ava'Hable |o them 
before this. Because of this new. information, they should 
gain new insights into their problems; hopefully, innovative 
approaches to the solution of these problems will result. 

Implications of the Model 

. The assessment model has potential for promoting cur- 
riculum development. This is especially true when it is ap* 



plied to state situations in the manner used in Colorado. In 
this situation, objectives were developed chat^specifically ap- 
plied to the local situatiqn. -^The statement of well-written 
objectives in- behavioral terms may sharpen the purposes of 
instruction. Througji the experience of wiping behavioral 
objectives, the curriculum worker^^gakw a^much (jlearer 
perception of his task; hepce, this practice mjrf^c^ave a 
' beneficial impact on curriculum work. On the other hand, 
the use of behavioral objectives has not always been a 
positive influence. The objectives may zero in on easily 
defined behaviors that lack scope and significance. They . 
may produce tunnel vision and put stV^ss on the inconse- 
quential and trivial. In an effort to be specific and to define 
the exact behaviors desired, the larger perspective may be 
lost. \ 

Again, the ^development of exercises from the identified 
* behavioral objectives may have a positive influence off cur- 
riculum. The kind of innovative exercises that have been 
developed by National Assessment may have a very positive 
influence on what is being ^lUght and how it is being taught. 
Teachers, both in reviewing exercises that have been used in 
National Assessment and in writing exercises for loca^^asses- 
sipents, may be influenced in their selection of both content 
and methods bycJheir knowledge of these assessment exer- 
cises. Material not relevant to the objectives of the course 
may be dropped, and methodologies promoting the kind of 
skills needed in the assessment exercises may be introdqced. 

On the other han^, the results may be less desiriable. If in 
state situations the dictates' of finances or the lack of 
leadership result in the use of poorly wriitten, machine- 
scored, merely multiple-choice exerciser, the results may be 
very negative. Teachers may feel pressured to stress rote 
learning of facts in order to prepare their students for poorly 
writUl^^^exHrninations. Hence, poorly written exercises may 
keep irrelevant material in the curriculum and limit cur- 
riculum innovation and development. The quality of the ex- 
ercises written and released will have an impaef'on cur- 
riculum development. 

Sampling procedures may give insight into the status of 
knowledge, understandings, skills and attitudes of students 
in a particular target population. Findings from these 
procedures can promote curriculum improvement and in- 
novation. Problem areas in the curriculum may be -iden- 
tified. Results of the National Assessment iVi citizenship 
have identified some problem areas. On an exercise dealing 
with freedom of speech, a large percentage of 13 and 17- 
ycar-olds indicated that they would not allow sample con- 
troversial statements to be made on radio or television. V - 
This kind of response indicated a lack of understanding o^ 
valuing of the Constitutional right of freedom to express 
controversial or unpopular opinions. 

On the other hand, there are potential difficulties with as- 
sessment data that represent national levels of performance. 
Even though the data were not collected with this intention 
and were reported in census-Uke form, the, results of 
National AssessmerK are being treated like national norms. 
Several states have conducted their own assessments 
duplicating , the NAEP model so that they can make direct 
comparisons between their state results and the various 
national, regional and group results.There js the potential of 
groat mischief in this appVoach, for it may lead ^ unfair 
comparisons between groups, states and regions. In the as- 
sessment reports of some states, tables of percentages have 
been presented without rny interpretation or explanation. 



11 



^ 00024 



Sopic school systems have been presented in a very bad way 
without any reference being made to the kinds of vkriabies 
involved in the difTcrent learning situations. Such vanebles 
as per'pupil expenditure^ educational level of parents and " • 
motivation^ of pupils do have^an ^pact on the learning 
situation. These and other variables should not be ignored 
in iijicrpreting assessment results. 

It is-not •Suggested here that assessments should nQ| be — ^ 
6onducted because of the potential misuses of the data. 
* fnstead, it is suggested that those engaged in assessment at 
national, state and Jpcal levels have the responsibility to. 
report data in proff^ perspective and to aid those using the 
data^.to make .correct interpretations of it. 'We need these 
kinds 6f information for decision making, but if the data ar€l 
misused or misinterpreted, then the decisions based on them 
may not be good ones. 

0 

Impact on Curriculum 

National Assessment is providing data on which decisions 
can be^made. The reports ori science and citizenship have 
resulted in strong recommendations for curriculum changes 
in these learning areas. As data are gathered at the state 
level through the use of the model and its adaptations, 
specifiCsuggestions for changes may be made. For example, 
Texas l\as designed its state assessment so that there is 
direct feedback at the classroo^'m level. This may have a 
strong 'and immediate impact on these classrooms, either 
good or bad depending on what interpretations and recom- 
mendations are made. 

.Still other problenjs should be pointed out concerning 
curriculum decision making and national and state assess- 
ment results. While the National Assessment process for 
identifying objectives provides for broad-based participation 
in the decision-making groups, objectives still are selected 
that neglect sizable, sub-cultural groups in oiir nation. Even 
some of the very general objectives selected may not apply to 
these groups, and thusd they are not valid for some 
situations. ^.^ To illustrate the potential* difficulty with* objec- 
tives and groups, consider Objective V of social studies: 
"Have a reasoned commkment {o the values that sustain a 
fr^e society." Under it is subpoint B, "Believe. in the rule of 
law and can justify their*belief." For blacks in the core city, 
this objective may not carry value. Their experiences may* 
have been that the law is used against4hem, and they con- 
AStantly need to be looking for new ways to work around the 
law.*" While this is perhaps an extreme example, it does 
make the poiiit that in as diverse a population as we have in 
the United States, there will be many groups for whom the 
broad general objectives do not carry the same meaning as 
for other members of the nation. 

There is also the question of what the components of a 
general education are or should be. Selecting common ob- 
jectives for a learning area sncfi as science and writing exer- 
cises for these objectives provide one definition of what stu* 
dents of a certain age level are expected to know about 
science. Since t-his establishes what comprises general 
education in science, it has a definite limiting impact on a 
student's freedom of choice in deciding what he wants to 
learn,. He is being dictated to concerning what he should 
take in general education. This situation always has existed 
in education, but it puts the assessment movement in the 
camp of the conservatives in the current controversy with 
respect to free choice and unlimited electives for students. 
Here the assessment movement is counter to the humaniz- 



ing"^ fT)ovement in American education. It is promoting a 
closed rather than an open approach to ci^rriculum. 

The very identification of a learning area for National As- 
sessment will have an impact on the fortunes of that learning 
area. The "chosen" learning areas are more likely to be sup- . 
ported financially and retained in the curriculum than those 
not selected by National Assessment. Those learning areas 
that are not included in the "magic" 10 lyiay find that they 
are seconci'^lass as far as school boards are concerned. If the 
students of the district have made a poor showing on the 
state ^assessment, funds may be shifted to the support of 
those sutfjects where the low scores were identified. The 
learning areas that are not getting this public exposure may 
find their financial support reduced. While the potential for 
better education decision making is here, there is also the 
potential for poorer education del:ision making because of 
the impact of public exposure of the assessment results on 
the decisionmakers. The foreign languages are not included 
in the 10 learning areas assessed;' hence, language depart- 
ments will not be able to cite assessment data as evidence 
that there is need for greater support of their programs. 
They have been put into a poorer bargaining position by 
.this omission. 

Moreover, there potential for "shortcut" assessment 
schemes by publishers, although as far as this writer knows, 
no assessment tinstrument of thi^ type has been produced to 
date. Why go to all of the work for an expensive local Assess- 
ment effort when you can buy a commercial assessment 
package, which is based on the released NAEP exercises and 
whitfi will provide the school district with results that may 
be compared 'to NAEP results (norms)? The exercises may 
be given and scored by the local teachers, and the results 
may«be compared with the National Assessment results iji- 
cluded with the commercial assessment package. If the pro- 
ject is handled "right," a "live-wire" superintendent can 
demonstrate at a modest cost that his district is outscoring 
the National Assessment results (norms). 

The idea of a commercial assessment package based on 
released NAEP exercises has positive potential. With 
honesty and careful application, this approach could be of 
considerable value to school districts. First,, the objectives 
being assessed by the exercises«included in the instrument 
need to be carefully identified. Then if the school district 
finds that these objectives fit their own objectives ade- 
quately, the exercises may be used wi^h confidence. Second, 
there is.no reason why teachers should not be able to con- 
duct and score the exercises accurately. A tape-paced ad- 
ministratioa could be used, and the directions for scoring 
could be written in such a way that teachers could follow 
them with good results. District-collected data about the 
performance of students on NAEP exercise's could be 
valuable data, if collected at a modest cost. There are a good 
many "iCs" and pitfalls in this proposal, but it is true that 
accurate, valid data could be collected in this fashion. Un- 
doubtedly, some districts will find this to be' a workable 
plan. 

(>ost is one of the big problems. It took a great deal of 
money to develop National Assessment's sophisticated 
program. Currently, the data being provided by the Assess- 
ment are of high quality, and the results have great promise 
for promoting education improvements. Nevertheless, 
hundreds of local districts are considering their own assess- 
ments, and the question of how to reduce the cost is becom- 
ing important. .Should they develop their own assessment in- 
struments, or should they use a commercial version of the 
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model? For many districts, it is not possible to develop their 
own* assessment programs; hence, for many reasons, the sec-* 
ono alternative will be used. Over the coiHiing years, the 
commercial assessment package will probably become a 
reality.* The only other alternative open to poorly financed 
districts is to resist the pressure to become involved jl^ local 
assejssn 2nt. 

Another potential outcome of an assessment program is 
the spin-off of research projects designed to investigate ques- 
tions of causality raised by assessment results. It has been il- 
lustrated several times in this chapter that^further investiga- 
, tion is needed to interpret results more productively. Any 
number of potential doctoral studies in NAEP data need to 
be made before the results may be confidently used in cur- 
riculum work. 0 

A final inter<psting prospect in «this entire assessment 
business is the increased pace of change in our society 
within re-^ent decades. Thosr of u«i who have been involved 
in education decisions over the last couple of decades know 
that ca|chin<; the direction of things is frequently more im- 
portant than trying to make careful, data-based decisions. 
By the time we have collected and analyzed our data base, 
society and technology have gone off and left us. It is hard to 
fault the soundness of the data-collection design of National 
Assessment, but there is a five-year time lapse from start to 
finish of a cycle, plus' the time lag of scoring, and data 
analysis. The question is whether this is the best process for 
education decision making in the last quarter of this cen- 
tury. Are changes in our society coming so fast that long 
before the data base is established the data are no longer 
relevant for the decisions for which they were gathered? 

Use of the Model for 
Accountability Purposes 

.Assessment is not the same thing as accountability, for ac- 
countability places greater <;paphasis on value judgment 
. than assessment does.^^ Accountability is 'concerned with 
the badness or goodness of someth^'^g. Education assess- 
ment is aimed at improving decision making by collecting 
inforrp^ition concerned with the outcomes of education. Ac- 
countability, has varying meanings depending on who is 
writing about it. . 

A number of approaches have been proposed to mhke 
schools more accountable, such as a systenrjs approach, 
management by objectives, education-prografn auditing, a 
planning-programmitig-budgeting system, performance 
contracting, vowcheV plans and alternative forms of 
education.^'* A widely accepted interpretation is that ac- 
countabilfty is to determine whether the teacher, who h as- 
signed the task'of educating a group of students, is perform- . 
ing that task.^'^ On the other hand, a broader interpretation 
of accountability' is that it is a process for determining 
whether the program of a school, district or state is produc- 
ing the student achievement expected with regard to the ob- 
jectives of the program.^" The first of these interpretations 
puts the responsibility for individual pupil achievement on 
the teacher; The ^econd places the responsibility for the out- 
put of a program of instruetioft on the school, district or 
state school system. 

The first internfetatjon oF accountability has been widely 
debated and hj^ gained the opposition of many groups in- 
cluding the teachers' organizations. A model of this ap' 
proach to accountability follows. 



I. Behavioral Objectives 
II. Stated Evalifa^o^K^riteria Related to the Objectives 

III. Scheduled iMaterials, Learning Activities, Equip* 

ment. Etc. 

IV. Teaching Activities 

V. Evaluation of Student Performan(fc Based ^ 

behavioral Objectives 
If the use of this model stresses the output of the teacheh 
in terms of pupil achievement, it does not provide for the 
multitude of variables that arc found in any teaching situa- 
tion. The primary emphasis becomes "Are tfce children 
learning what the teacher was hired to teach them?" No al- 
lowances are made for the many variables such as pupil 
ability, parental education or wealth of the school district, 
all of which and more may infiuence the success of the 
teaching in a given learning situation. - 

Under this approach to accountability, complete data ' 
must be collected for each individual in the population. It is 
necessary to identify the performance of specific individuals 
with respect to stated objectives since the responsibility for 
the individual student's performance is to be assigned to the 
instructor charged with teaching for these objectives. Here, 
it would seem, accountability is taking us back to yearly, 
mass testing with which some of us are only too familicir. 
Test administration procedures, such as NAEP's tapepaced 
method, could be put to work, although, because of financial 
considerations, teachers would prqbably administer these 
mass tests with many of the problems in the results that 
have emerged in the past. , 

The second interpretation of accountability presented 
here also sees where students are or are not achieving, but it 
is concerned with what the strengths and weaknesses of a 
program are and how they relate to student achic^ments. 
The NAEP model may easily be adapted for this type of ac- 
countability use. In assessment, the objectives areidentified, 
the achievement level with respect to these objectives in a 
population is determined through a sampling 'i^^sessment 
procedure, and the results are reported in terms of what 
percentages of the^pdpulation are achieving the objectivds. 

The National A«s«sment model, as it was originally 
designed, did not provide uSable flata -for determining who 
was responsible for the individual student either achieving 
or not |ichieving the stated objectives. As designed, the 
model provided information about the achievement or\ the 
stated objectives of a population or groups of that popirfi- - 
tion; hen^, it was not possible to idehtify the results 'for in- 
dividuals in these population groups. Neither, was it possi- 
ble to establish what individual teacher was responsiblc^or 
the students having eithe. achieved or hot- achieved |;he - 
stated objectives. 

Now let us examine the National Assessment model for its j 
application to the evaluation of an entire curriculum for ac- \ 
countability purposes. The model cpmpbnents identified in 
this paper are: objectives, development, exercise develop- /j 
ment, sampling plan, administration of exercises, scoring 
and analysis, reporting and dissemination, and utilization. 
Under accountability, objectives may be developed with in- 
put from a number of sources, including the patrons who are 
paying the bill for education, or an already established set of • 
objectives may be used.""- Since :hi's ia the evaluation of a 
given curriculum, the objectives should apply to that 
program if the results are to be. valid. Who should select 
these objectives is an issue in accountability, Shouldn*t there 
be input here from the teachers who are presenting tfte 
program ? 
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While {here arc some problei^is in how objectives are to be 
seleqed, the cd|tiponents,of the NAEP model are all func- 
iioneA m thi* ^adaptation for accountability purposes. 
Measurement" "^ecialists contend that objective-referenced 
exercises written for specific learning objectives are better 
for use- in accountability evaluation than, the norm- 
referenced exercises commonly used in standardized 
tests.-" Therefore, released objective-referenced, National 
Assessment exercises may be used for accountability pur- 
poses where their objectives are valid for the learning situa- 
tion that is being evaluated. 

^ In the National Assessment design, the samj^s v^ere 
drawn in such a fashion that they represented tj^e pop^la- > 
tion of 9, 13, 17 and 26^j5-year-olds and gftlips from these 
populations. Where the rriodel is being used to evaluate, thf * 
performance o\ a given population, sampling procedures, 
may be used as they were developed for the Assessment * 
model. Yhc same profession^! care that is tak^n in exercise 
administration of the NAEP model would be desirable in 
the application of the model for accountability purposes. 
Scoring and. reporting are the same for assessment and ac- 
countability, and the same perxenta^e kind of reporting can 
be used inr both situations. 

Where accountability is being applied to a total organiza- 

(lon such as a school, a district, or a state, the National As- 
essment model may be used with little modification. It was 
designed to establish accurately the level of performance on 
a given set of objectives in a population, and it can do this 
for accountability purposes as well as assessment purposes. 
Likewise, it can assess groups of the population- W that 
specific strengths or weaknesses in their perforrpance can be 
identified. / 

There are many value questions related to accountability, 
and while it is not the task of this report to discuss them, a 
number are 'cited here. Will tight state accountability struc- 
tures Severely limit creativity and innovation in the schools? 




0 

• As indicated in some of the previous comments, doesn't ac- 
countability have the potential of becoming a stultifying 
state-testing program? Will the single-minded pursuit of 
achieving performance objectives lead to the abuse of 
children? Cou'dn't a disproportionate amount of time be ex- 
pended x>n the defining of objectives wShout .a commen- 
.surate increase in learning? Could not objectives established 
by politically oriented groups set schools on a coiTse of in- 
doctrination? Currently, in education there- is « humanist- . 
behaviorist conflict. Doesn't the accountability movement 
support a kind of tcchno-urban fascism i^-* Could not the ac- 
countability fnovemrnt lead to the teaching of easily quan- 
tifiable material and discourage the inclusion of material 
^ .that is diffrtult to quantify? Doesn't accountability promote 
-a closed rafher^than an^open education system? Doesn't ac- 
■ countability present obstacles to the continued development 
of freedom and autonomy for teachers?, This is not an ex- 

r* haustive list, but these ace alt disturbing questions. Furtlier- 
more, there is considerable evidence In the literature that 
supports the seriousness of the issues raised by these ques- 
tions/*^ ^" 

Summary 

Several summary statements can be made. The National 
Assessment model is no doubt the be^t and most com- 
prehensive procedure designed for collecting d'ata for these 
purposes. The model can^^iijdhas been successfully adapted 
for use at tl)e state level; anda^ states become more active in 
assessment, many adaptations of it will be made to fit local 
needs. Finally, where the 'concern is with the evaluation of 
ejroup performance, the model may be used for accoun- 
tability purposes. With the current rapid development of 
state assessment and accountability progranTs. it is expected 
that ihc model will be widely used for these jD^irpqses in the 
coming years. \^^\ 



NOTES 

* 

If* 

1. Sec Appendix D, Reference 16 , . 

2- .See Appendix D, Reference 1 . • 
3. .See Appendix* D. Reference 2 

4 Vincent N. CampbelKand Dary! G. Nichols, National Assessment of Citizenship Education." Soual Edmatum 
U (June 1969), pp. 279.81. 
^ 5 See Appendix O, Reference 5 ^ 

6. See Appendix D, Reference 16. 

7. See Appendix D, Reference 9. 
8 See Appendix D, Reference 7. 

9. .See Appendix D. Reference 13. 

10. .See .Appendix D, Reference 1. 



ERIC 



14 



11. Sec Append s D» Reference 16. ' 

12. See Appendix D. Reference 9. ^ ' • ^ 

13. Sec A|3pen<iix H, Rcfer<ince 1^. 

14. See Appendix D, Reference 9. 

15. See Appendix D, Reference 16» 

16. See Appendix D, ^Reference 5. 

17. See Appendix D, Reference 9. 

18. Sec Appendix D, Reference 16. 

19. Larry E. Conway, "Some Implications of the National Assessment Mode! and Data for State and Local 
Education" (paper presented at the 1973 Annual Meeting of the AERA, New Orleans, February 26, 1973). 

20. See Appendix D, Reference 4. ^ • 

21. See Appendix D, Reference 3. 

22. Joan S. Beers and Paul B. Campbell, "Statewide Educational Assessment,*' StaU Educational Assessment 
Ptograms, 1973 Rtvision (Princeton: Educational Testing Service, 1973), p. 1. . 

23. Ibid. . . - ' , • 

24. Ibid . pp. 2-3. ' 

25. Maine Assessment of Educational Progress: Methodology y Report 5 (Augusta: Department of Educational and 
Cuhural Services, 1972). , ^ 

26. Report on the Assessment of Reading $kills of Connecticut f\iblic School Students Philadelphia: Institute for the Study of 
Inquiring Systems, and Hartford: Connecticut Department of Education, 1972). c 

27. Sixth'drade Mathematics: A Needs Assessment Report (Austin: Texas Education Agency, 1972). 

28. John VV. Helper, An Assessment of Learner J^eeds in Col^ado (Denver: Colorado Department o( Education, 1972). 

29. Joan S Beers and Paul B. Campbell, "Statewide Educational Assessment," State Educational Assessment 
programs, 1973 Revision (Princeton: Educational Testing Service, 1973), p. 3. 

30. Frank B. Wonder, Developing a Large Scale Assessment Program (Denver: (imperative Accountability Project, 
I9-?3). p. 89. . 

31. Marjorie M Mastie and Frank B. Womer, "How Wiil National Assessment Change American Education?" 
Phi Delta Kappan. 53 (October 1971), pp. 118-20. 

.^2 See Appendix D, R^erence 5, p, 34. 

33. Charies H. Harrison, "Are We Educating for Tomorrow?" Scholastic Teacher, September 21, 1970, pp. 16-19. 
."^4. J. Wayne Wrightstone et al., "Accountability in Education and Measurement Problems," Test Service Notebook 
a (New York: Harcourt, Brace, Javanovich, Inc.), p. 3. 

.See .Appendix D, Reference 14.- 
36. Stephen P. Henclcy, "Impediments to Accountability," Administrator's Notebook, Vol. XX (Chicago: University 
of Chicago, December 1971). 

37 Frank B. Womer, Developing a Large Scale AssessrtUnt Program (Denver: Cooperative ^Accountability Project, 
1973), p. y 

38. Lesley H. Browder Jr., An Administrator *s Handbook on Educational Accountability (Arlington, Va.: Arperican As- 
sociation of School Administrators, 1973), pp. 14-21. 

^9. J Wayne Wrightstone et al., "Accountability in Education and Measurement Problems," Tes^Service Notebook 
ii (New York: Harcourt, Brace, Javanovich, Inc.), p. 1. 

Lee J. Cronbach, "Evaluation for Course Improvement," V?Mi/in^j in Measurement and Evaluation, ed. Norman E. 
Groniund (New York: Macmillan Company, 1968), p. 41. 

41. Lesley H. Browder Jr., An Administrator's Handbook on Educational Accountability (Arlington, Va.: American As- 
sociation of School Administrators, 1973), p. 50. , 

42. Irving Morrissett, "Accountability, Needs Assessment, and Social Studies," Social Education, 37 (April 1973), 

274. ^ 

43 J Wayne Wrightstone et al., "Accountability in Education and Measurement Vrob\ems,'' Test Service Notebook 
U (New York: Harcourt, Brace, Javanovich, Inc.), p. 5. 

44. Stephen P. Hcncley, " Impfediments to Accountability, "^i/minir/rfl/or*jA^o/?ftoo*^Vol. XX (Chicago: University 
of Chicago, December 1971). 

45. M.M. Gubser, "Accountability As a Smoke Screen for Political Indoctrination in Arizona," Phi Delta Kappan, 
55 (.September 1973), 64-65. 

4r> Harold C H a nd.>" National Assessment Viewed As the Camel's Nose," Phi Delta Kappan, 54 (April 1973), 539- 
41. 

4? Ernest R. House, "The Price of Productivity: Who Pays?" Today's Educattgn, 62 (September 1973), 65-69. 
48. Jacob Landers, "Accountability and Progress by Nomenclature: Old Ideas in New Bottles," Phi Delta Kappan, 
?4 (April 1973), 539-41 

49 Julian C Ornstein and Harriet Taimage, "The Rhetoric and Realities of Accountability, " Today's Education, 62 

tt,^ .\ . L . _ • nr* \ *?AQ A 



\ 



15 



00028 



CHAPTER 3 



* Q 



ON THE NEED FOR CRITERION-REFERENCED 
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MODEL OF NATIONAL ASSESSMENT IN CITIZENSHIP 
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Teachers College, Columbia University 



I 

In this paper I want to underscore first of all two related 
meanings of the term ''model," one emphasizing'^the struc- 
ture of what is represented by a model, and the other stress- 
ing the phenomenon of influenc^ Next, I would like to note 
the potential interplay between these two aspects of the 
National Assessment of Educational Progress (NAEP) 
. nriodel. with reference particularly to possible interpreta- 
tions of the data in Cittzemhtp Reports 2 and Last I shall 
suggest an alternative to National Assessment, taking note 
of the gains the project may have ijiade, but at the same time 
orienting stiil more directly to political andeducation goals, 
in this case to the fostering of citizenship. 

The National Assessment Model 
as Representation and as Influence 

This paper will dwell upon the obvious, and th^ distinc- 
tion to be made hfcre is no exception. Taylor's chapter 
(Chapter 2) describing the NAEP model and its uses itself 
points to both the repres'^ntation and influence aspects of 
modeling, thus my starting point is onq of clarifying what he 
has written. 

When we speak of the ''model'* of National Assessment 
we refer partly to some representation of its basic compo- 
nents and their articulation.. Exhibit 1 in Chapter 2 
describes a succession of steps in the development, ad- 
ministration and dissemination of NAEP instruments and 
findings. The multiple connections drawn^in the diagram 
suggest complications in this process, as the first-cycle 
sampling plan was^modJfied in anticipation of scoring and 
analysis, exercise development looked forward to the utiliza- 
tion of information, etc., and as these steps might be 
repealed in successive cycles. 

Exhibit I spells out, or differentiates, only steps of special 
concern to the assessor. A different diagram might elaborate 
the process between dissemination, utilization and the 
redevelopment of objectives, suggesting different channels of 
national, state and local dissemination; different 'uses of the 
assessment inff^rmatinn^o^ fMP^tif>n ofn^ilTerent resources 
-Tit--rfTe5e"lcveTs; and so on. Taylor's diagram reflects the 
putative neutrality of National Assessment vis-a-vis the uses 
of the assessment information. In various passages Taylor 
does suggest that this information calls for research, that it 



will be interpreted differently at state and local levels and 
that it will lead to interventions in different spheres — 
economic, curricular and so forth. Still, it is the basic stance 
of NAEP that these things be left to others to work out. 
From this standpoint Exhibit 1 represents the Assessment 
'process accurately. 

But a model is never neutral. Whether it has been drawn 
in physics or in politics and education, a model modth a 
process of thought and action. In this simplistic way one can 
argue that what National Assessment models is assessment, and 
that complementary processes directed to changing what is 
assessed will be more an indirect, than a direct, result of 
National Assessment. It is not a trivial observation that the 
uses of the Assessment have in fact been, as described by 
Taylor, further assessment at the state and local levtls. 
NARP*s powerful coordination t)f money and technical ex- 
pertise with academic and political judgment partly ac- 
counts for this imitation. Equally, however, it is the 
specificity of the model with' respect to certain processes, 
and its lack of specificity with regard to others, that ac- 
counts for the imitation. Indeed the A-ssessment itself can 
be said to be an imitation of a process that has been most in- 
Huential in Aiucrican education in recent decades, the 
testing of individuals for purposes that are a great deal more 
diffusp than the testing operations per se} The criterion, 
referencing of the National Assessment exercises might be 
seen as a move beyond testing for its own sake, or beyond 
the parallel purposes of grading and selection associated 
with norm-referenced tests. Whether this move is realized, 
however, and the testing thus does not again become an end 
in itself, depends on other processes that the NAEP model 
dors not evnliratr M nr^Q^nt Tf ;iQc^Qom<«nr i*o onr^^^.^J • - u« 

' i — f- r - - IV i^x. 

linked more closely to teachers* accountability than it has 
been in the past, there remains the risk that accountability 
will be displaced upon the population tested. 

Let me now repeat this argument a. anothcr level of what 
is madQtedby jVational Assessment ^ its representation of 
what citizenship is and how one goes about observing and 
explaining its occurrence or nonoccurrence. Reasoning 
simplistically again, let us first suppose that acceptable per- 
formances on the citizenship exercises are the criteria of 
citizenship — i.e., that we would sa/ someone is a good 
^ citizen if she/he performs in acceptable ways on the test, ^ 
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analogously to our saying that someone is a good driver if 
she/he passes a driving test.^ How does NAEP allow us to 
explain the occurrence or nonoccurrence of these pcrfor- 
mance^r?. Age, sex, parental education, color an<i location by 
region or size and type of community arc the only variables 
that are systeniatically broi^ht to bear on this question. 
One might stirmise that age represents different degrees of 
exposure to citizenship education, but identifying whether it 
is explicU or implicit, or whether it stems from school, televi- 
sion or community, defi|cs conjecture except on a most ad 
hoc, exercise-by-exerciJe' basis. Sex might relate to role op- 
portunity, parenfftl education to kinds of discourse in the 
home, color to an individual's treatment by others and loca- 
tion to* community needs and resources; but then we could 
scramble all of these conjectures top, given the lack of 
specification of what the variables link with. What we are 
led to, then, if only by^default, is an emphasis on the popula- 
. tion characteristics themselves as explaining acceptable and 
unacceptable citizenship performances. 

Caplan and Nelson^ recently reported that, of 69 data- 
based, psychological research studied of blacks that were 
abstracted in the first six months of the 1970 Psychological 
Abstracts, 82% were person-centered and only 18% situation- 
or environment-centered in their \:ausal attributions. By far 
the largest category of variables Caplan and Nelson in- 
cluded under the heading "persoo-centered" was "group 
membership (e.g., black or white),'* which included 48% of 
ail the 69 studies. Person-centred categories, the a:jithors 
observe, are more available and more easily exploited than 
situational or environrnental categories. Discussing a variety 
of questions such as achievement motivation v. the structure 
of the economy as causes of unemployment, Csfplan and 
Nelson concluded: 

Whether the social problem lo be attacked is delin- 
quency, mental health, drug abuse, unemployment, 
ghetto riots or whatever, the significance of the defin- 
ing process is the same: the action (or inaction) taken 
will depend largely -on whether causes are seen as 
residing within individuals or in the environment.^ 
Are we not dealing with a similar phenomenon? The ef- 
fect of not specifying what age, sex, parental education, 
4 color and location mean experieniially is our attribution of 
good^and bad citizenship to these qualities f>er se. 

The National Assessment citizenship reports do speculate 
at certain points about the meaning of the population 
variables. The following excerpts from Citizenship Report P il- 
lustrate the tentativeness with which this is done: 

Thirteen-year-olds in the grade school group 
showed the greatest deficit of any group — 10% — on 
the racial-attitude exercises. Their responses to 
several questions describe where some of the deficits 
occurred, but iWh*t explain why. . . . Perhaps as 
higher and higher proportions of the population get a 
hi^h school education, the smaller the proportion who 
do not become more distinctive in certain ways (e.g., 
less acceptinc; of otYier races).*' 

Are black youngsters less willing to tell a nonblack 
interviewer what they believe about racial dis- 
crimination? Or are they really less aware of- racial 
discrimination than other 1 3-year-oIds? The balanced 
results ort these questions show smaller deficits (by at 
^ ?a5t halO than do the observed results, as we discus^ 
sed earlier in thi5j:hapter (see page 51). At least one 
of the other characteristics on which the resuhs are 
balanced is thus associated in some way with the 
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resuhs reported for blacks. For example, perhaps the 
disproportionate number ^f blacks whose parents 
have little education hear jess discussion about acts of 
racial discrimination in the world.' 
These and a dozen similarly limited speculations aside, 
the main tendency in the reports is to leave the findings to 
explain themselves. Presumably research to be designed and 
^conducted by other agencies will lead to the development of ^ 
interventions that will in turn result in changes in the assess- 
ment results the next time around. What may defeat this, 
however, is the "psycho-logic" of the model. The structural 
emphasis of the inodel on population characteristics, rather 
than on the interactions 'of individuals with different en- 
vironments, induces thinking in these same stereotyping 
terms by those who receive the assessment reports. 

The language of thejiTtizenship reports consistently rein- 
forces stereotyping: 

The two upper levels of parental education, par- 
ticularly the beyond high school group, excelled, as 
usual.^ 

The typical performance of blacks at all four ages 
shows deficits of about 9% on all citizenship resuhs 
combined.'^ 

The general picture is for performance on this goal 
to follow that on all citizenship resuhs fairly closely. 
Thus the extreme, rural and extreme inner city 
respondents showed the greatest deficit in relation to 
the nation as a whole, and the extreme affluent sub- 
urb respondents showed the greatest advantage. . . 
Of course, this i^ all very ironic. The reports feature as 
many exceptions to the general trends as they can find, and 
the avoidance of discussing the significance of the general 
trends is presumably calculated not to risk offen?e. But the 
pptterrtS in the test resuhs that are associated with the pop- 
ulation variables are clues to individuals' interactions with 
their environmentsrThat our thinking cannot afford to rest 
with these clues^is the thrust of the next section of this paper. 

The last point to be made in this opening argument 
returns to the analogy between a citizenship test and a driv- 
ing test. SimpUstically again, we have^ to notice that 
National Assessment presents what individuals jay in 
response to various paper and pencil, interview and discus- 
sion tasks as the basic model of wfiat consCitutes citizenship. 
Who can doubt that many more individuals know or will say 
that they should vote than -actually vote in any election in 
this country -i- national, state or local? Or that more believe 
they should oppose discrimination in a park, and can say 
how to do so, than are likely to put this belief and knowledge 
into practice? Kohlberg'^ has emphasized Hartshorn and 
May's classic failure to find differences between delin- 
quents* and nondelinqucnts* knov^ledge of '*right" and 
"wrong** actions and has stressed instead the developmental 
level of the individual's justification of right and wrong. 
Kohlberg thus improves his capability of associating moral 
discourse with moral behavior, and, analogously, wc niight 
come closer in this way to associating citizenship discourse 
with citizenship behavior. The developmental approach, 
however, still leaves questioiVst^unanswered. Is it possible 
that those with **higher'* levels of mor^ discourse are more 
capable of rationalizing their morality and immorality, and 
thereby more readily escape being branded delinquent or 
criminal? Our increasing awareness of *'white collar" or 
corporation and political crime certainly gives credence to 
this question. Again, is it possible that different styles of 
language and discourse associated with socioeconomic or 
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racial/cultural differences affect the response of an in- 
dividual to a vert£al representation of a moral, or let us say a 
civic, problem and affect equally an e5;?miner*s interpreta- 
tion of the response? Hgfe the questions we arc raising with 
relation to what is known or suspected about moral judg-^ 
ment lead into the more general area of performance in a 
culturally standard verbal situation, in this case the formats 
that link testing and schooling so closely with edch other. 
Labov's'2 finding that the task of talking to keep a rabbit 
from getting nervous elicited much mofe talk from children 
who spoke black English than questions addressed to the 
children by a sympathetic black ijnterviewer illustrates this 
concern. Could all this have any bearing on who^ tends to 
"exhibit** less "knowledge and behavior considered 
desirable for citizens in our society"? Suppose the test was 
one of integrating a neighborhood, a bus or a lunch counter, 
or that it was challenging a school bureaucracy, or that it in- 
volved hiring or working or playing or generally living with 
someone who had been in jail, as opposed to saying how one 
would do any of these things? 

Pan of what is at stake here is the relevance of the 
knowledge required by the assessment to the particular con- 
texts in which different individuals enact their citizenship. 
This question applies not only to, say, who is more likely to 
be oriented to the courts as an institution for settling dis- 
putes about money, but who is more likey to have to accept 
doctors and dentists.of a different color from one's self. (Just 
to indicate that the question can cut both ways!) What I 
would emphasize here is that the Assessment is more likely 
to have conteht validity to the extent that it includes the 
enactment of citizenship in goal-related contexts. This then 
connects with the other part of what is at issue, the format 
or structure of the setting in which citizenship is to be 
observed, which.on linguistic grounds alone, as I have tried 
to argue above, is more likely to be valid as it is oriented to a 
cteeoship goal — i.e., to a citizenship goal other than ac- 
ceptable performance on an exercise — not to mention how 
the Assessments Jiaving acceptable and unacceptable 
responses for each and every exercise contributes to the 
cultural standardization of "advantages** and "deficits.** 

All of the above might seem to show thatUmsK jio_use for 
informatioin about what individuals say, or that I think dif- 
ferences would vanish if enactment, instead of saying, was 
assessed.'^ctually, I would like to know about both enact- 
ment and saying, or the knowledge and feeling saying repre- 
sents. The National Assessment exercises do in fact include ' 
self-reports of enactments, but, even accepting the reliability 
of self-reports, these are still no more revealing in and of 
themselves than are sheer statements of knowledge and feel- 
ing. How knowledge and feeliiig fadlitate or inhibit enact- 
ment is a question that especially concerns educators, along, 
with how situational conditions facilitate or inhibit enact- 
ment. Docs the civics text knowledge represented by many 
of the exercises contribute to elTeclive action in some 

I .v..«^.w...^ MOV* III wtllUJIS. V^I J lU 

rephrase a different question asked in Citizenship Report P, 
what rules irake most sense to different individuals par- 
ticiftating in a ^ommon tas^? 

Let us state the basic question still more generally. It is 
not who are the good citizens, but what are the conditions 
that contribute to good citizenship. Because NAEP is not 
designed to answer this question, however much it may 
speculate about it ex post facto, it docs not model asking the 
questton. We will now proceed further into the logic and 
"psychologic'* of the problem this presents. 



The Meaning of Patterns In 
National Assessment Findings 

National Assessment reports tend to discuss the explana- 
tions of specific anomolies associated with one or another 
exercise, but they avoid discussing the meaning of the larger 
patterns that hit the reader full in the face. Here we shall 
take note of some of the difficulties these larger patterns 
present. Our approach to this will be naive. We shall first 
orient ourselves to the dat4 itsdf, asking what the patterns 
might be likely to suggest to the reader. Then we sha*! just 
begin to sort out some of th^ factors that are confused in 
these patterns, raising questions for an alternative program 
of research and demonstration. 

Age as a Variable in Citizpnship Performance 

\n Citizenship Report 2, straight percentages of correct or 
acceptable re^onses at different age levels arc shown for 
caLh exercise reported on. Thus the following results arc ex- 
hibited: ^ 

Report that the police do not Adult 
have the right to come inside 
one*s house at.any time they 
want and can give as a 
reason legal guarantees, or 
reasons concerning privacy 
and permission *of occupant 
to enter(in own words). 



20% 68% 90% 83% 



State that our legal system 
(courts, lavvs) is the means 
provided by government for 
settling an argument over 



money. 


50 


70 


87 


Last names of the persons now 


1 






holding these offices 








President (Nixon) : 91 


94 


-97 


98 


Vice President (Agnew)'^ 


60 


75 


87 


Could give aTleast4^cxplana- 








tion of y/hat fighting was 








about in country named 


53 


66 


77 


2 explanations 


27 


44 


55 


3 explanations 


11 


24 


31 


4 explanations 


3 


12 


16 


5 explanations'^^ 


1 


4 


7 



Opportunity to read a greater 
variety of viewpoints and in- 
formation was stat^rf as a 
reason why it might be good 
to have newsfiigpers in a city 
written and printed by more 
than one company.'^ 



37 64 88 



92 



As these examples illustrate, the general tendency in the 
data is for the percentage of acceptable responses to increase 
with age, though with some reversals between age 17 and 
adult. This trend is alluded to in various specific contexts in 
Citizenship Report 2, for example in the following comment: 
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As seen above, adults consistently showed more • 
knowledge about current conflicts than did the other 
ages. When asked in Exercise F4 to name some ways 
to avoid war, however, fewer adults than 13s and 17s 
named at least one way (77% of 13s, 8S% of 17s and 
65% of adults).^ Even 9-ycar-olds approached the 
adult achievement, Jevel, 60% giving at least one way 

• to avoid war fEjjtercise F3). One explanation is that 
the adult view pf the possibility of avoiding conflict is 
both more sophisticated and more ^^aundiced'Y a 
larger number'of adults indicated that they felt war to 
be unavoidable ^nd a larger number at the younger 
ages gave simplistic, but acceptable, answers ("stop 
fighting''). 

The number who gave at least three ways to avoid 
war demonstrates ^he more usual age trend of an in* 
crease in achievement u^ to age 17 with minor dif- 

• ferences between 17s and adults (5% of 9s, 12% of 13s, 
32% of 17s and 25% of adults). A similar 9-to-13 age 
trend is shown in Exercise F5, which supposes com' 
petition between the U.S. and Russia concerning ter- 
ritorial rights on Mars. AWhost twice as many 13s as 
9s (75% V. 45%) statc^hat the U.S. and Russia 
should discuss and s^^e these matters before men 
land on Mars.'** . 

Notice that in the discussion above, as is the case 
elsewhere in the report, it is the departure from the "more 
usual age trend'' that seems to call for explanation. How 
does one account for the main trend itself? Leaving aside for 
the present a consideration of how the exercises were con- 
structed in relation to age expectations, and assuming the 
content or goal-referenced validity of the exercises, the con- 
clusion that best fits the pattern in the results is that they 
reflect maturation and/or the general accumulation of 
knowledge and experience with age. This is obvious, is it 
not? 

What is less obvious is the significance of the conclusion 
for citizenship education. Many, but probably not all, 
readers of this report will be aware that social studies 
educators in the 1950s (I choose this date for convenience of 
discussion) had occasion to assess the significance of similar 
findings arising out of a number of surveys of children's 
social concepts conducted during the Progressive Education 
era: Meltzer, 1925; Lacey, 1932; Pressey, 1934; Eskrid^C^ 
1939; Ordan,i945; and Bates, 1947 — to name those I am 
most fan^iliar with.'*^ All of these studies included elemen- 
tary school pupils at different grades, and several of them in- 
cluded secondary school pupils as well. All tested qhildren's 
attainment of the correct or conventional meanings of social! 
concepts at different ages, and all found progressions in cor- 
rect attainment with indrtasirig age. Interestingly, only 
elementary social studies educators, with the exception of 
Wesley and Wronski^" (Wesley himself was author of a 
*Test of Social Terms" in 1931^^), paid attention to these 
fliiuings, directly or indirectly, tc judge frcn* «^ ccmpsnsors 
of elementary and secondary social studies methods texts of 
the 1950s. At least in the decade before the impact of Sput- 
nik, Bruner and the structure movement in curriculum, it 
was elementary school educators who were more concerned 
with the development of knowledge ov^r time or age, while 
secondary' school educators focused on its acquisition ai a 
given time or age. 

But which emphasis was correct? The^lemcntary school 
educators^tr::sscd a progression of learning from concrete to 
abstract, the necessity of building up manifold experiences 



with the referents of a concept, and the importance of 
teachers' avoiding empty verbalisms and premature =*for- ^ 
malization. What precipitated out .of this, however, was 
such an emphasis on accommodating to the presutned , 
"natural" *pace of concept attainment reflected in 
progressions of attainment with age, that Bruner 's declarj^- 
tion than anything could be taught in some honest way a^ 
any age, hedged though it was in Piagetian conditions, came, 
as a shock, particularly to elementary school educators. \ 

None of the social-concept studies cited aboVc, nor any 
others conducted in the years before 1950, tried to assess 
children's capability of learning a concept at a given age by 
systematically attempting to teach thfc concept to children. 
Lacey did interpret' an observed acceleration of concept^ 
learning at grade 3'as being the result of a more deliberately 
planned social curriculum than in grades 1 and 2. Though I 
do not know them, therc.niay well have been many studies 
carried out during thes«! years that -weighed the relative 
merits of one method of teaching social concepts versus 
another as a variety of more recent studies have done.^^ 
Who, however, has set out to teach social 'concepts to a 
criterion at a given age, i.s., to teach until the. pupils reached 
the criterion, and in this way Studied the methods that 
resulted in different children's Ifcarning at that age? We may 
often teach with reference to a criterion; but in the regular 
or the experimental social studies or civiqs classroom we 
typically abandon a unit of ifikiruction to m6ve on to 
another unit or to^discontinue an experiment, while sOme» 
pupils, at least, still have not met what we might, hold as 
even a minimum criterion. Of course, this f^ises all kinds qf 
other questions^ about the desirability of convergent versus 
divergent learnings, etc., but that is beside the point here. 
What I sense is that there are a variety of factors, perhaps 
especially in social studies and civics, that contribute to a 
criterion-referenced system devolving .into a norm- 
referenced system- One of these factors is the very 
knowledge that individuals' differences with respect to at- 
tainment of a criterion held to be.**reasonable" for a given 
age will tend to diminish as more of these indwiduals attain 
the criterion past that age. 

Without understanding tlie issue completely, I think there 
is an ambiguity in National Assessment's own position on 
this matter. On the one hand, we have been. told that **out- 
standing local teachers familiar with each target age group 
(ages 9, 13^ 17, adult) worked for weeks with our staff to 
break down each general objective in the most germane 
behaviors appropriate as goals for a given age group. "'^ On 
the other hand, wc learn that one criterion for exercise 
development was that "some exercises cover important at- 
tainments which nearly everyone is successfully achieving, 
some which very few people are achieving and some which a 
middling number achieve."'^ Does the latter aim, together 
with the finding that Assessment performances did in fact 
break dovyn this way for each major citizenship goal,'' imply 
th3t such 3 distribution should continue he nhtainrH in 

future assessments? Perhaps the aim oT NAEP is to provide 
information that will encourage state and local striving 
toward full attainment of the goals for all individuals. 
Taylor, however, has observ<?d a tendent / for Assessment 
data to be treated as a national norm against which both 
state and local test performances are being cunipaicu (see 
Chapter 2). Given the statistical patterns within and 
between the age levels that National Assessment models, I 
think this is inevitable.'*' 

National Assessment findings are likely to be taken as 



20 



developmentaf norms unless several steps to counter this are 
» taken. (1) The exe'rcises would have to be broken down into 

their more truly developmental and nondevelopmental com* 
ponents; (2) the assignment of exercises to age .levels wQuld 
need to be done from the standpoint of what it is 
theoretically possible to expect any educable individual of a 
givei^ level to achieve; and (3) experiments would have to be 
conducted to demonstrate that 100% of the educable in- 
dividuals at an age level could achieve acceptable perfor- 
mances for the exercises at that level 

This is a rather drastic prescription, with^'many problems 
inherent in it.. Its function in this argument is to underscore 
the fact, recognized by NAEP but likely to be overshadowed 
by the pseudodevelopmental patterns in its fmdings, that 
National Assessment so far has not tried to take cognizance 
of what has in fact been deliberately taught, specifically in 
the schools,' thus its findings should not be construed as sug- 
gesting limits to what individuals could be taught^ or could 
learn, at the different age levels of the Assessment. 

But wc did admit that this has its problems. One of these 
is distinguishing what we called above more "truly" 
devt^lopmental from nondevejop mental components of 
learning, it seems to me, for example, that any 9-year-old 
could learn that policemen do not have the right of un- 
restricted entry to ^home. At the same time, 9-year-olds' 
reasons for such a belief would less sophisticated than 13- 
year-olds' reasons, focusing at the first level on, perhaps, the 
more direct consequences to the persons involved, and at the 
second on more general properties of the social system like 
the, rights of individuals to privacy and the effects of not 
regulating entry into homes on other spheres of the social 
' system, instruction might then concentrate on all^ 9-year- 
olds attaini.ig at least the first developmental level of 
justification^ and on all 13-year-olds attaining at least the se- 
cond level. Of course, in addition to the questions already 
raised above about the significance of different developmen- 
tal levels* further questions would now come into play about 
the efficacy of direct instruction toward developmental 
goals. Still, this woul3 represent a considerable shift in the 
validity of the developmental problem. 

Another problem stems from the question of whether a 
criterion in citizenship is more validly held for individuals or 
for the society as a whole. Consider thr case for the latter 
standard first. Almond and Verba^** have suggested that a 
political system might get overheated if too many of its 
citizens participated actively in the process of governance, 
beyond voting, and thus that there might be optimal levels 
of le^s than full participation for the stability of even a 
democratic system of government. (Almond and Verba 
werei indeed, rationalizing the levels of participation they 
. found in American society.) Apart from whethe. or ijot one 
r agrees with the value of stability in the system, this does lead 
one to distinguish between citizenship goals for the nation or 
' _ foi^ a region as a whole and goals for all the citizens of the na- 

4 ti9n or a region. Maybe only a few arc needed to, for exam- 

ple, come up with mamy ways that war could be avoided or 
that discrimination in a park could be stopped. When 
CHizenship. Report 2 referred to "goals that only a select few 
were achieving, "^"^ I wondered if a part of what this implied 
was that only a fevv are needed t;o achieve these goals. 

The trouble with this reasoning, however, is that the 
many who do not achieve the goals in question may include 
precisely those whose interests are most at stake, viz., those 
most likely to be frontline soldiers in a war or most likely to 
be excluded from a park. Once we allow that less than all 

\ 
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need to achieve one goal or another for the health of xYit 
body politic as a whole, we run the risk of playing into just 
those features of the system that presumably account for 
some groups in the poptklation being consistently "disad- 
vantaged" no matter what the goal in qu^stipn is. Further- 
more, many of.the goals set by National Assessment pertain 
to the rights and obligations of individuals in their everyday 
relationships to one another and to the law. Anyone of a 
relevant age who does not know that the coUrtS are available 
for the resolution of a dispute over money might be disad- 
vantaged in the event of such a dispute. Similarly, anyone 
who discriminates against another^on the basis of his or her 
group identification contributes tojthe denial of the rights of 
the other to be treated as an indiviBu^l. 

A third problem arises from the fact that, especially if, in 
line with the position advocated earlier, instruction with 
respect to a particular goal at a given age was pursued until 
all individuals so instructed reachedthe goal criterion^ there 
might not be time enough to reach all the goals held to be 
desirable for individuals at that age. 
^ Still another problem is that it cannot be assumed that 
school is the most appropriate setting in which to intervene 
to try to bring individuals to a criterion. But then there is no 
way of knowing from the present design of NAEP what part 
the schools have played in the performances on almost any 
-of the assessment exercises. Even specifying which criteria 
the schools could effect would be progress from this stand* * 
point. Especially for those, criteria that we have 
characterized as "enactments," it might* be that school 
wouIg have to be coordinated with other agencies or institu- 
tionsv family, local government, citizens groups, television 
and so on. National Assessment may assume that efforts of 
this sort will be an outgrowth of the publication of assess- 
ment results. It would be a ^ore likely outcome if the con- 
tribution of such efforts to the attainment of goals was 
specified. 

What all of this reasonmg drives me to, then, is a different 
kind of neutrality from that which is modeled by NAEP. 
Instead of publishing results that encourage normative 
thinking in the face of virtually no knowledge of the condi- 
tions of individual citizenship learjjiing and development, an 
alternative strategy should concentrate on specifying the 
conditions .that make full attainment possible for one or 
another criterion at a given age level. Tlie freedom of state 
and local agencies or institutions to emphasize those gpals 
that mattered most to them would thqs be an enabling one, 
rather than the spurious freedom that sheer ignorance af- ' 
fords us. 

Sex, Parental Education, Chlor and Location 
by Size and Type of Community (STOC) 
as Variables in Citizensfiip Performance 

The logic with respect to these variables is essentially the 
same as it has been above, though I will be repeating it in ' 
somewhat different terms. First, let us attend again to the 
"psychologic" of the matter. The very familiarity of 
National Assessment findings, especially with respect to 
parental education, color and location, may tend to give 
tliem a normative significance, i.e., to imply that the ob- 
tained patterns are what we should continue to expect. For . 
me, at least, this'stereotyping is reinforced by the technique 
of comparing group performance levels with national levels, 
rather than with the criterion of 100%, so that there will 
regularly be groups with "advantages'* and groups with 
"deficits," relative to each other (shades of norm referencing?), 
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even when, say, it is 60% of one group axid 50% of the other 
that has reached a criterion. Be that as it ^ may, the 
familiarity oftheresuhs that I am referring to is their com- 
parability to the findings of any number of assessments, 
from the Army Alpha intelligence testing program that 
might be sai^ to have started it all, to the Coleman Report 
^ and still more recent studies. The sheer accumulation of 
these findings Has contributed to their reification in recent 
years, so that in the Jencks analysis socioeconomic status 
and color appear to be almost intractable constraints on 
education, or at least on education by the schools. 

Let me focus, this thinking in terms of the Colemvi and 
Jencks analyses, raising a few questions at a very superficial 
level. First, these studies base their (somewhat differing) 
conclusions about the limited capacities of schooling to af- 
fect achievement on achievement defined in very close rela- 
tion to general cognitive skill, vyith all of the cultural 
loadings that the phrase "general cognitive^ skill" implies. 
As Jencks himself observes, the Clolemaii^tests of Verbal 
Ability, Nonverbal Ability, Reading Comprehension, 
Matlfcmatics Achievement and General Information inter- 
correlated so highly that "the student who did \yell on one 
test and poorly on another was quite extfcpt^ionan **'** Assum- 
ing for the Purposes of this argundcnt that socioeconomic 
and cuhural laetors reflected, in the school will lai%ely deter- 
mine the school's effect on general cognitive skill, it rgiiains 
possible that schooling can have a more independent effect ' 
in areas of achievement that arc more specific to school in- 
struction and, convcfselWles& generally diffused in the . 
society at larg^The Intd|j|gtjonal Education Assessment's 
(lEA) current research on factors affecting achievement sug- 
gests, for exampid, that the effectiveness of science instruc- 
tion is substantially more independent of nonschool factors 
than is the effect ivc^icss of irtStruction in reading^ or 
arithmetic.*^ Pursuing the principle that might explainmiis, 
we first have to account for lEA's finding that their/Social 
Studies/Civics Education test results beha^ more like 
reading and arithmetic than like science! 

The lEA researchers themselves attribute their findings to 
the permeation of citizenship knowledge, like the three R's, 
throughout life outside the schools. It makes sense from this 
standpofqt that the citizenship results would reflect varying 
access to, a^dhccasion or power to use, the knovs^dge that 
the tests irfCWporate. But is not science also diffused 
throughout life in a modern society? Of course this is the 
case. One must reason then thafscience education and the 
science test exercises have been couched at a higher or more 
specialized level of knowledge than is reflected in everyday 
life, and that citizenship education and/or the citizenship 
test exercises have not been defined at the samc^relatively 
high or specialized level. Studies of the learning of higher 
mathematics, as* opposed to the common mathematics the 
Coleman (re-analyzed by Jencks) and lEA studies vvere con- 
cerned with, do show that differences in schooling make a 

lA/U-v. : 1 » . 
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effect ip^ citizenship. 

What might be meant by "higher" or "more specialized'* 
citizenship knowledge? One need not think of it simply as, 
say, secondary school or cojlegc level information and con- 
cepts, but rather as knowfcdge that is couched at a higher or 
more complex level thayscems to be reflected in citizenship 
achievement at any g^n age under conditions other than 
systematic instruction. Thus one would aim tc teach at a 
given age what NAEP or some other survey shows is not 



"normally" achieved at that age, or perhaps not even at a 
subsequent age. v , 

Another, complementary way of tjonstruin^ what might 
be meant by higher or more specialized citizenship 
knowledge stresses, instead of • official or ideological 
doctrine, a more skeptical, probing interpretation rf)f 
citizenship. Neither NAEP nor lEA has included knowledge 
exercises of this sort, ahhough the lEA instruments do in- 
clude attitude or opinion items that question the economic 
and political systems of the society^ Can wc say that the ' 
schools should also aim to teach to individuals of a given 
social background that which is not ''normally" known to* 
persons of that background in our society, be they rich or 
poor, and should cultivate questioning, in that social sectoi; 
of a sort not "normally'' realized in that sector? 

Arnoff*s^- investigation of factors related to the ability of 
children in second, third and fourth grade \q comprehend 
concepts of government points in the first direction sug- 
gested above. Arnoff designed a five-week government cur- 
riculum to include (though not c;xclusively) concepts not or- 
dinarily included in instruction for th«e grades — ^ i.e., not 
included in social studies textbooks aTtncse grade levels. Ar- 
nofPs results clearly showed the effect of instruction. 
Seventy-five percent or more of second graders, fof example, 
learned 23 new concepts of local, state and hational govern^ 
ment: property tax, split ticket, subpoena, judge, etc., most 
• i^of which w€re not included in second'-grade social studie^ 
textbooks. Furthermore, social class tended not to correla^^ 
with more or less learning of new concepts in this experi- 
ment, thouglf mental age as defined by an intelligence test 
did correlate with new learning. \, ' 

A field trial of the American Political Behavior (APtiT 
course developed bv the High School Curriculum Center in 
Government at Ihdiana University^' points partly in the se- 
f? cond direction indicated above. The APB course aimed to 
teach 'Tacts and ideas about politics th^t Have not been part 
of typical social studies curricula, "^V for example, facts 
about the proportions of different income groups that vote in 
American elections. Differenqjs between experimental and 
control groups' ,pretest/posttest knowledge gains were 
clearly demonstrated in all nine communities involved in the 
field trigl. On the other hand, comparable differences in 
political- science skills achievement were demonstrated in 
only four of the nine communities, and effects on students' 
attitudes were demonstrated in none of these communities. 
Furthermore, while there was some variation of student 
background characteristics, the communities involved did 
not include the rural and inner city extremes identified in- 
National Assessment; and almost all of the students were 
white. > 

One should not suppose that systematic instruction could 
eliminate the effects of poptilation or student background 
characteristics, or indeecjl that this adequately represents 
what is desirable. The effects that National Assessment 
makes us principally aware of are those that stem from 
restrictions on different groups* access to and control of in- 
formation; if citizenship education should aim to minimize 
these effects, still this would surely entail interventions out- 
side the schools, and research and demonstration would 
need to be directed this way as well. At the same time, the 
schools must recognize the identities and priorities of dif- 
ferent social groups; paradoxicaUy, whatever equali?ation 
the schools accomplished with respect to the distribution of 
information and skills might contribute in some ways to 
heightened affective differences between groups. 
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A second problem in the Coleman study, one that the 
Jencks analysis only partially rectiiies and that ^fational As- 
.scssment simply ignores, is the question of comparability of 
school environments, in terms other than socioeconomic 
status (SES), between and within difTerent locations. It is 
well known that Coleman compared whole systems as to 
library size p^ir-pupil- expenditure, level Of teachers* formal 
education and so forth, whereas these conditions might well 
vary between schools within a system. Jencks used Project 
Talent data to make these comparisons between individual 
high schools, and still obtained no effect independent of 
nonschool factor^. The lEA science flndings do show such an 
effect, not for all the variables that might be thought to be 
relevant, but for *^the opportunity to learn, the student time 
in hours per week and cumulative years, the curriculum 
emphasis and the additio^l years^ of postsecondary 
preparation of teachers. In none of these analyses has 
classroom climate or methodology seemed to account for 
diOerences in students* achievemeni^. 

My own position is that one must compare not just 
programs or schools or even classrcAms in order to specify 
the environmental Conditions that affect schooling's con- 
triblkion to achievement. Rather, it is necessary to examine 
different settings wilhin classrooms, or within school-related 
locations outside the classroom, in order to nnuike headway 
with the questidh of educational envtronments.^^ In my ap- 
proach to this problem over the past few years, I have found 
it useful to distinguish between high teacher control, joint 
teacher and learner control and high learner control of each 
of various conditions within one or another classroom set- 
ting j(or subsetting) and thus to" be able to ask hpw much 
time the students in a given classroom spend under different 
conditions of control. Task options, palling, teacher-learner 
and learner^learner interaction, teacher adaptivencss and 
task-performance criteria are some of the conditions of set- 
tings that can be observed to vary with respect to control. 
So-called **open classrooms,'* for example, can be seen to 
vary among themselves \n the proportions of time that stu- 
dents spend under these different conditions of control, 
whether one is comparing time for aggregates of students' 
bet,ween two or more classrooms or is comparing time fbr in- 
dividual pijpils within a single classroom. 

This afSproach further distinguishes between different 
types of education purposes: the transn^issioi^ of knowledge, 
tradition and experience to all learners in a setting alike; the 
operationalization of partly commcto) but also partly in^ 
dividuated, competencies or skills s^lid concepts; and the 
cultivation of individual and collaborative venture, or ex- 
ploration, problem solving and expressive composition or 
construction. Intellectual, socidl^ physical and emotional 
aspects of learning and developmenlf are equally involved in 
each of these categories of purposes* , 

I have hypothesized both that the effectiveness of an 
educational setting will be partially dependerk on the inter- 
nal consistency or congru^ncy of its controls, knd that each 
broad type of education purpose will be realized best in a 
diffcrcnj^typc of setting, viz., transmission in a high teacher- 
control setting, operationalization in a joint teacher- and 
learner-control setting and venture in a high-learner control 
setting. It is apparent^ then, that I believe that what has 
variously been called "classroom ^limate** or "classroom 
method" will someday be understood to relate to achieve- 
ment<) or to type of achievement, even though the results of 
research on this to date have been very confusing. 



This report iias emphasized education in the schools^ but 
it has recognized at more than one juncture that going . 
beyond Earner or citizen population variables to the interac- 
tions of individuals with their environments would entail 
research and demonstration outside of the school as well. 
What are the effects on individuals' acknowledpnent of 
racial discrimination, of this acknowledgment *s actually 
contributing or not to ameliorating the discrimination? How 
is one*s orientation to a legal system for settling an argu- 

• ment over money affected by one*s having, or anticipa^ting 
having, money enough to invoke the legLl system in a dis- 

' pute? Does involvement in the affairs of a rcspoi;isive public 
agency or institution lead tH'greater participation in other 
public or community affairs? 

My paradigm for these questions, and many more, is 
Kohn*s'^'' research on the conditions that determine fathers' 
values fok their children. Social dass, defined as oc*^ 
cupational position and education, does relate in Kohn's 
data to whether parents tend to value self-direction or con- 
formity to external authority — higher social class being as- 
sociated more with the first and lower sociabclass nriore with 
the second of these value clusters. VVhen the analysis con- 
trols for the degree of self-direction or autonomy ^tln^t the 
fathers experience at work, however, social clau differences 
tend to vanish, lower-class fathers w'ho experience 
autonomy in their jobs valuing autonomy for their children 
^ as much as middle-class fathers who experience this 
autonomy do. In^ other words, while Social class is an ap- 
proximation to the conditions that determine fathers ' values 
lor their children, the experience of fathers in their work 
describes these; conditions more exactly. Is it not possible 
that research could similarly track down conditions as* 
sociated with, but still independent of, sex, parental educa- 
tion, color and location that would more exactly explain the 
associations of citizenship achievement with these variables? 
Further research could then be^ directed toward changing 
these conditions experimentally, so that, just as we might 
demonstrate in what way achievemient other than that 
which is "normally" associated with different age levels is' 
possible, likewise we could show how achievement other 
than th'^t "normally" associated %vith different sex, parental 
education, color and location statuses is possible. 

An Alternative to the National 
Assessment Model 

From various directions, our argument converges on the 
desirability of emphasizing person-environment interactions 
and demonstrating the achievement that is possible under 
varying person-environment conditions. Consider one 
further vantage point that ^Jiffers somewhat from ours so 
far. Etzioni," in a discussion of organizational analysis, 
criticizes that assessment of organizations that focuses on 
goal attainment. "One of the major shortcomings of the goal 
model is that it frequently makes the studies' findings < 







tions."*'' An-organization frequently does not reach its goals 
effectively and often has goals other than the ones it claims 
to have. It would be more useful, Etzioni concludes, to treat 
goals as cuhural entities in themselves and to ask how 
various internal an/d external conditions contribute to the 
realization of observed goals. 

I do think NAEP's attempt to define citizenship goals and 
to describtf^f^rformances that represent the achievement of 
these goaft^t^ valuable ^e fact that National Assessmerit 
anticipate modifying ae goals and exercises to reflect 
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chjangcs in priorities between assessment administrations 
indiqaies at least some awareness of the goals as ^'cultural 
entities." But what is served by publishing only those state- 
ments of goals that represent ihc '^'consensus" of those con- 
sulted? Wquld not the publication of disagreemcnts^over the 
definition of goals, particularly disagreements stemming 
from different subcultural interj&retations of citizenship, ^cld 
further to the demythification and destereotyping of 
citizenship goals? 

^, What stands in the way of this process is, once again, the 
Assessment model, its structure and assumptions. Let rfine 

. claiin intuitively that a national test of citizenship simply 
cannot Ije consistent with cultural and political pluralism. 
National Assessment is first ^ind fOrefnost a test and as such 
depends upon conformity to its ^oais for its authority. 

An alternative to National Assessment would conjiuct 
research and demonstration in specific relation ig 
citizenship goals but would recognize that these goals have 
different meanings and priorities for different individuals 
and groups. The altcrr|ative should, indeed, explore these 
differences explicitly, as^n understanding.of them would be 
essential to any application of the research and demonstra- 
tion. , ' r i 
- What I have in mind first of all*, then, is a pluralism of in- 
quiries to replace the monolithic National Assessp.iem, in- 
quiries that could be drawn upon differently by different 
at^rncies or groups gr rather, since anything can be drawn 
upon in this way up to a point, that encouraged different agen- 
cies or groups to order their efforts according to what is most 
meaningful and important to them. ' 

An alternative to National Assessment should be truly 

' criterion oriented. Its aim should be to use analysis and in- 
tervWition to bring the observation of performances as cjose 
as po^Hible to criterion, allowing that different agencies and 
groups wiH- draw differently on this research anddemonstra- 
tion in subsequent applications of it. 

Many separate studies would be involved, but their intent 
would be the same. The basic paradigm would be that 
which shows convergence on the achievement of goals as the 
resuft of reducing differences between groups, or aggregates, 

' of different statuses, by specifying the conditions associat,ed- 
with these statuses that originally account for the differences 
in goal achievement. Is the seemingly lesser awareness of 
racial discrimination on the part of blacks a result of unwil- 
lingness to dis<*lose this awareness in a test or interview? 
Then perhaps the difference would diminish as observations 
were conducted in settings in which the res^ndents felt 
more in control Are parents of lesser educational attain- 
ment less involved in the politics o/ their ch.ldren's schools 
be^'ause of a feeling that school had not been responsive to 
them as children? Comparing groups on the basis of this 
feeling might reduce the difference attributable to status and 
intervention through the schools to respond to alienated 
parents might increase the parents' involvement in then* 
children's schools. > > 

Probably hundreds of studies that throw light on the 
specific conditions contributing to citizenship have already 
been reported. These could be inde?iied by goals, and again 
by conditions, so as to make this knowledge available "to 
policymakers at different levels. ,Somewhat in cjcample of 
this is a document prepared by rhc Social Research Group 
at ITie (ieorge Washington University, Research Prohlems^and 
Issues in the Area of Socmlization, 1972, part of which analyzes 
what is known about the development, the determinants 
and the changing of intefgroup and intragroup attitudes 
and behaviors. Of course, such an analysis points as well to 
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what is not known. The following quotation from the Social 
Research Group's report represents a juncture that .is fre- 
quently arrived at in their analysis; 

The significance of some of these findings on 
cooperation i^ far from clear. We still do not know 
how cooperation determines intergroup attitudes and 
behaviors or its role as a factor in achieving a success- 
ful ethnic and social class mix. What does seem clear 
. from the research* is that cooperation sometimes leads 
to better intergroup relations, although much work is 
needed to, determine the conditions under which 
cooperation produces an enhancing effect.^" 
"••Much work docs need to be done. Some of it consists .pf 
formal skidies in which the investigator attempts to control 
the principal variables and to predict outcomes precisely in 
advance. Often, however, these studies are too rigid to be 
able to deal with the unintended factors that enter into every 
complex action — which may partly accpunt for formal 
studies failing to confirm their hypotheses. Action^research, 
in which the action is typically guided by more evolutionary 
goals and in vwhich "real time feedback Wcontinuously 
regulates the participants' activity in relation to goals, com- 
pensates in fiexibility for lvhat it may lose in control. A for- 
midable agenda of both fprmal and action studies, then, 
must be undertaken, all of it referenced with respect to goals 
and conditions alike. ^ 

' Ferhaf)s the case could be made that developing a 
knowledge base for citizenship education in this way would 
-still require a determination of national performance levels 
as in NAF,P. In my opinion, however, this' would be dis- 
tracting at the least, and possibly destructive. An alternative 
wav of construing the national" significance of a study 
would be that it deals with problems that are best ap- 
f)roa( hed with national resources, while a more local study 
deais with problems for which local resources are adequate, 
or, to put it differently, for which only local resources are 
likely to be appropriate. Thus a study could have national 
significance even if it did not have a national sample (which 
^is not the same as saying it would not be carried out in a 
variety of communities). The effects of income maintenance 
on citizenship performance might be construed as a national 
quesyon because only the federal government has the 
money to maintain incomes. The effects on citizenship per- 
^ formance of collaboration between schools and other,conrv 
munity institutions or agencies might be construed as a local 
\ question because only local individuals or groups have the 
power to bring about this collaboration. 

C'learly a citizenship- research and demonstration 
program could go on forever, insofar as our awareness of 
what'we do not know multiplies at the same rate, at least, as 
does our knowledge, Then National Assessment equally 
could contij^e forever, since the goals it sets will probably 
change in lil^^ manner. Substantial human and material 
resources have been invested in NAEP but this is not the 
only, or even the chief reason, to think that the project will 
be continued. The technology of the Assessment, as of 
testing more generally, peculiarly lends itself to the collusion 
' of academic ingenuity :and political decision making. Were 
the questions this paper raises foreseen? Probably some pf 
them 'were, and others were not. If the present form of 
.National Assessment is itself the result of compromises 
meant to render the project less dangerous politically, this 
does not bode too well for the alternative that I propose. On 
the other hand, what are the political consequences of what 
National Assessment hath wrought? These too will have to 
enter in^o the equation. 

• 
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CHAPTER 4 

EVALUATING SOCIAL STUDIES AND CITIZENSHIP EDUCATION: 
SOME ALTERNATE APPROACHES 

0 

Michael Scriven 
University of California, Berkeley 



There is tremendous overlap between the areas of social 
studies and citizenship education — well-illustrated by the 
i overlap in the goals and objectives developed by the two dif- 
ferent institutions, the Educational Testing Service (ETS) 
and the American Institutes* for Research (AIR)i that con- 
tracted with thv* National Assessment of Educational 
Progress (NAEP) to develop specific statements. I shall not 
make any great efTort to separate them^^^-here because our 
concern is with evaluation models^and only secondarily with 
specific content — and very similar problems arise ^n both 
area'f. 

4-ct's call the NAEP approach, well-described by Bob 
Taylor, the first approach to evaluating social studies educa- 
tion. I shall describe three othrr approaches very briefly and 
suggest a synthesis. Hien I shall look in soiriewhat more 
detail at certain features of the National Assessment ap- 
proach and- Bob Taylor *s conr.ments. 1*11 begin by focusing 
on the area of citizenship values, understanding of law and 
due 'process, etc., because it*s more important than 
geography and in worse shape. > ^ 

Alternatives to National Assessment 

A second possibility would be a comparative approach in 
which direct international comparisons, Tor example, were 
sought. This process would require a substantfal but not 
complete revision of^the \Xtxii pool in order to make really 
direct comparisons; but some direct and many indirect ones 
can be made using the data we do have from the Institute of 
' Inr^rnational Education studies and others. These com- 
parisons are extremely inriportant — even more than they 
were in the math studies, for example — in showing' what 
/>ar« Arxrxf* oc wcl! 3S ^H^t ^° beifi^ T^* citszd«s>^ip 
/social studies areas are not so abstruse that one can 
reasonably suppose them (or" most of them) to be beyond the 
grasp ftf a substantial majority of pupils. Of course, there 
are important difTersnces between countries that would 
make it necessary to proceed with caution, in inferfing from 
what has been done elsewhere to what could be done h<?re; 
but the difTerences are nat such as to make attempts (ex- 
periments) absurd which is all we could justify to start with 
anyway. Of course, too,* tremendous chauvinism is as- 



sociated wit^^^the citizenship/iocial studies area, and there 
would be many who would condemn any attempt to match 
the performance of other countries per se. However, the 
merits of that argument seem as slight here a$ in, for exam- 
ple, the automobile or , He psychiatric or the adult education 
field. With General Motors switching to the Wafikel dngine, 
acknowledgedly copying Mercedes in styling and suspen- 
sion and Ferrari in design {vnth the 1975 models), one can 
hardily argue that the experts can't sec any transferability of 
foreign ideas. Similar cases are welUknown in the other 
fields mentioned. If we are interested in reducing the level 
of antisocial activities and basic ignorance about con- 
stitutional knd other rights, and ?bout humiin nature — 
which is what citizenship/social studies is all about — then 
it seems appropriate to look for possible improvements 
wherever we can Hnd them. There is plenty of evidence in 
the comparative education studies to date to suggest that we 
could do better, but we do need more precise comparisons. 
Telling us whether we're doing at well as we could is one of 
the functions of evaluation, and it isn't done very well by 
NAEP. 

Another type of comparison that would really be signifi* 
cant would involve comparisons between-the performance of 
pupils — still measured mostly on paper and pencil tests, at 
' in National Assessment — attending Schools with radically 
difierent approaches to citizenship/so<:ial studies. The con- 
trast between those with a conventional curriculum and 
those using some of the ahernative approaches — -e.g.. Pro- 
ject vSocial Studies materials — might be illuminating, and 
the discovery that there wasn't a contrast would also be il- 
luminating. Of course, the Assessment docs make Some 
comparisons, e.g., between performance of black and white 

^ i:**!-. u 1 s. 1* 1 t u . „ J 

to introduce new curricula or methods. One might put it 
fhis way: NAEP and most state assessment programs are 
pretty good photographers, but not very good buying 
guides. For that you need the relevant coVt.parisons, viz., 
those between the available options. 

A third possibility involves switching to a very difierent 
kind of item, albeit still a paper-and-pencil (or vocal) te$t. 
Here we'd go to something like the Social Issues Analysis 
Test, where the item might present a page-long newspaper 
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editorial or a dialogue and a series of rather searching ques- 
tions on iu of a wide range of difficufty. We might allow a lot 
of time for this perhaps an hour. An essential selection 
criterion /or items would be novelty: typically they wQuld be 
hypothetical cases to which the **approved" answer Jias not 
yet been identified by adults With whom the*pupils interact 
(to rule out parroting). The reasons for shifting to this kind 
of item are OTthat the few items like this that Rave been 
released reveal the most appalling incompetence in 
operating with the simplest constitutipnal or moral princi- 
ples (e.g., freedom of speech), and it is now most urgent to 
clarify the real situation; (2) although the analysis of 
responses would require formidable training and talent on 
the part of the scorers (since much reading Between the lines 
would be necessary) there would be a corresponding in- 
crease in the significance of the results. Instead of telling us 
where the pupils are at, this kind of test can tell us where 
they are capable of moving to, which is our only hope (given 
the abysiAal level of performance at the moment). The page 
of dialogue can involve argumentation and can call for 
evidence of understanding the steps in the argument and 
their effect on the reader (e.g., by using interspersed ques- 
tions and indelible markers, etc.). Despite the use of 
hypothetical situations, the responses are much more likely 
to be realistic here ;han in the present items, where the use 
of stereotypes by the student can easily provide a facade of 
answers that tell us nothing about the probable response to 
a new case. In short, bad though the present answers are, 
they may well give huge overestimates of the merit of the 
respondents^ which leads us back to reason (1) above. 

.\ somewhat radical extension of this approach leads to a 
fourth evaluation, which would move ihto the field and away 
from pencil-and-paper tests and use the best skills of the 
anthropologist and the sociologist (besides those of an ex- 
tremely acute content analyst) to identify the values of 
various age leVel$ and adults in our society from a study of 
their communications and decision, processes. To take an 
extreme, but extremely important, example r— during the 
hearings before tfie Ervin committee we were presented with 
a very detailed pitrturc of the level of moral analysis<:^nd 
citizenship behavior of the White House staff. The addition 
of the tapes has made this a very cpmplete data source for 
the kind of question I'm raising here. Similar analyses can 
be done of the discussion at the school board and in the local 
press of a proposed decriminalization Order to a city police 
department (and of subsequent events), or of the discussions 
in an eight-grade classroom of a proposal to vest disciplinary 
powers in the students (and of the subsequent events). 
These analyses are tricky; there are few analysts presently 
equipped to deal with them objectively — but ohj what a 
treasure trove for the evaluator is there! Here we can bypass 
the problem of test invalidity; here we are dealing with real 
actions and perceptions. Despite the massive media 
coverage of Watergate, I never saw any analysis of the 
<;ianifiranrr nf rhr ronrrntiorm revralrH bv HalHrman and 
Chrlichman on the stand. Most people got the feeling that 
they were ''sort of morally blind,*' that they were abusing 
their power. But consider Ehrlicnman's justification of the 
burglary of Dr. Fielding's office: **As we saw it, it was as if 
you/learned that a map was stored in the vault of a D.C. 
bank that showed the location of a bomb that would blow 
up the whole of the district the next day; wouidn^t^ou think 
it was justifiable to 'break and enter*?'* [paraphrase]. There 
were a dozen sirpilar examples. I think there is more infor- 
mation about tijk evaluation of citizenship/social, studies in 
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U.S. schools and homes in those passages than in all the cor- 
responding test results from National Assessment. This was 
not just one aberrant lawyer speaking. This was a line of 
argument — grotesquely irrelevant though it \/as — that 
readily persuaded almost everyone on whom it was tried by 
someone coming from the White House. We did not learn 
al out Watergate from someone who had a better education 
in citizenship/social studies than Ehrlichman; we learned 
about it from a black nightwatchman doing his job well — 
for which he was essentially blacklisted. 

There is a recurrent tone*in the Watergate discussions at 
every level — media. Congress and neighborhood — and the 
same note can be detected in the discussions of any other 
widely discussed moral issue of our time, such as drug law 
and enforcement^ "excess*! profits by oil companies,, etc. 
That tone is naivety — and from our point of^ view, par- 
ticularly naivete about the psychological nature of mankind, 
society and morality. We need more careful evaluation than 
we have yet had to determine whether this impressionistic 
reaction is ill-based or not. The third and fourth methods 
described here use simple enough procedures, which we 
have often applied in evaluating competency in other areas 
of interest, e.g., in testing cognitive, mechanical and ad- 
ministrative skills. I believe they deserve more serious ap- 
plication in citizenship/social studied, where we have so far 
— with regard to our own society. — alternated between 
oversimplified paper-and-pencil tests and overemotional 
social documentary. 

One feature of the field-study or anthropological ap- 
proach, which deserves some stress, is that it does not begin 
(or does not need to begin) with the massive effort involved 
in developing goals. There is something slightly inap- 
prop. iate about that effort for an evaluation task, it seems to 
me; it is exactly the right activity for developing a new cur- 
riculurfi, but'that is hardly what NAEP v^^as supposed to be 
up to. (It's perhaps not too surprising that considerable op- 
positio'.t to National Assessment arose from those who felt 
that it was attempting to impost a monolithic citizenship/ 
social studies curriculum on U.S. schools; the complaint 
might seem stupid at first glance but on second thought 
reflects some sensitivity to a significantly possible outcome, 
*v " school politics being what ihry are.) It seems plausible 
enough to argue that you ran\ set up tests until you know 
what they, are ifsis^ for, aiic whsJt t^iry are tests /or — i.e., the 
goals of citizenship/social studies education. But that*s an 
error, as we'll sec in the next section. Here 1*11 just stress the 
existence of an alternative approach. One could have had a 
team analyzing adult behavior in the citizenship area for 
deficiencies by\ identifying the optional feasible behavior in 
the situation in question and extracting the discrepancies. 
After a long search, one would then classify the discrepan- 
cies and set up the assessment program to determine the ex- 
tent of these deficiencies in the population. This involves no 
reference to the goals of citizenship/social studies education, 
fhniiffh qnrh roiilH hr infrrrrH from it; if shopf-rirriiit<j thai 
concept. ^ 

.So the three models I am/pmposing might be called the 
comparative^ the simulation And the anihropotogtcal models. 
They are mere sketches he/e, of course, but I believe thi/ do 
serve to open our minds /o the existence of rather different 
approaches to evaluation of citizenship/social studies 
education — possibly they will serve as useful targets for dis- 
cussion. We have become somewhat fixated on the "stan- 
dard" model of as?' 'sment, and* we have invested in it very 
heavily (see annual reports of Educational Testing Service 
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[ETS] Center for Statewide Assessment). I think we have 
become too rigid in using this model, and I see no reason 
why some diversion of resources could not be made to in- 
clude at least some of the other models I Have described. 

But that*s'not the only possible way to chaijgc. There are 
major changes ip the NAEP model that deserve considera- 
tion and that could also produce an ''alternate form," which 
a)u\d bp used alongside the continued use of the present 
forms (desirable for obvious reasons). We'll turn to these ir^ 
the next section. 

Changes in National As^es^ment 

A tremendous price was paid for '*political" acceptability 
of the* NAEP approach, and this may well have been the 
right decision. However, there is some.point in talking about 
ideal ways of evaluating, and even the feasibility question is 
probal^Iy due for reconsideration. The two'big trade ofls (or 
sellouts, depending on how radical one feels this mornint^-) 
are: 

1. Restriction of goals to those '^accepted as an 
educational task by the school,"' or "acceptable to most 
educators and considered desirable teaching goals in most 
schools.'" A further restriction was to goals that were '^con- 
sidered desirable by thoughtful lay citizens."^ 

2. Restriction of items t6 those that most states liked. In 
some cases, it is clear, particular states would not accept 
certain items in the version of the test forms used within 
the'r boundaries, ahd to avoid becoming widespread, com- 
promises had to be made on other items. Of course, 
citizensliip/social studies were the areas hit hardest by this 
constraint. 

These are serious limitations indeed. If schools and states 
can vote on the standards by which they are to be judged, 
^e are simply going to lose some very fundamental 
criticisms. The process actually gave the veto power to each 
of three groups — schol.irs, educators and lay people. 
That's a pretty tough obstacle race for an objective to get 
through and some pretty crucial ones didn't make it, es- 
pecially — unfortunately — those that would niost acutely 
test the moral sensitivity of students on controversial issues. 

These restrictions might be relaxed after new consulta- 
tions, or they might be bypassed using the field-study ap- 
proach described earlier. One way or another, they are bar- 
riers to a full evaluation. ' . 

One does not judge the education of lawyers or doctors by 
asking the law' school or medical school for criteria (or let- 
ting them veto external lists). One judges it by a careful 
analysis of thfe performance of the professionals in the field, 
lising the testimony of clients and coworkers who see that 
work — an analysis that looks not only at deficiencies, 
which will always be with us, but also at the question of 
whether these deficiencies are the kind that could have been 
removed by education, preferably an education that is 
fiscally and temporally realistic. 

I he schools are permeated by a number ofumfortunate 
ideologies in the citizenship/social studies area, ndcologies 
that are tremendously destructive to reasonable citizenship/ 
social studies education, and completely fallacious. Un the 
light of these ideologies, educators reject certain kinds of 
goals for citizenship/social studies; yet these include many 
goals of the greatest importance. A couple of examples may . 
suffice to illustrate the point. The fact-value distinction, and • 
the associated ideology of value-free science, is pervasive 
among educators (and many scholars and thoughtful lay 



people). Hence they will not accept goals that assert the ob- 
jectivity and factuality of certain moral standards, and the 
falsity of others. Indeed they go further and require (i.e., did 
accept) objectives like SSIIA, 17-A: "Distinguish among 
definitional, value, and factual issues in a dispute."^ Thiols, 
of course, the chin edge of the relativistic wedge. If one can*t 
say that it's a fact that Ehrlichman improperly approached 
Judge $yrne, then ethics is indeed a travesty; but of course ik 
is a value claim ^nd if these are exclusive categories, it can't 
be both at once. 

There are other glaring omissions in the objective lists 
concerning the foundations of ethics, the relation of ethics to 
religion, to conscience, to the law, to custom and conven- 
tion, to pragmatic considerations — the very issues on 
which a person's ethical commitment founders in the 
tempest of a personal crisis. But as my second example, let 
me take something less philosophical, more specific — the 
understanding of Communism. Is there a more important 
issue? Is there a worse-taught issue? Is there an issue on 
which we need information more desperately? Are there 
searching questions aimed at discovering true under- 
standing rather than slogan memorizing? Clearly not. Here 
is a case where the label on the package will pass the 
educators ("teaching about Communism" is an acceptable 
goal), but the only sane way to do it (use Communist docu- 
ments and speakers, [live, taped or filmed) as well as critical 
commentaries) is entirely unacceptable. The same applies 
to homosexuality, adultery, prostitution, violence, abortion, 
pornography, etc. — in short, to most of the topics that are 
likely to produce ^ personal moral crisis for the graduate of, 
or pupil in, our schools and that can be thoroughly and 
helpfully discussed there. Instead, they have to be discussed 
by the \Valking wounded in later life, too late for primary 
prophylaxis. 

The second major weakness in the NAEP approach lies in 
thexonceptualization of the goals and objectives. NJ/ithout 
detracting from the very considerable merits of ETS and 
.MR, who did the work, the goals and objectives leave a 
' great deal to be desired and bear the heavy signs of comng^it-. 
tee authorship. A few examples from the citizenship goals 
will indicate the kind of problem that exists. 

(jbal A ik "Show concern for the welfare and dignity of 
others."^ Of course, showing concern is not what we want; 
we want having concern. It's attractive to go for the 
**behavioral objective" formulation, but it focuses on exter- 
nal signs when we want something much deeper. Someone 
who does not show concern but who gets the ambulance is 
better than someone who weeps hysterically. 

Objective G-I is "Try to inform themselves on socially 
important matters and to understand alternative view- 
points."'' Is the goal trying? Or Is the goal succeeding? S\i\i\ios^ 
you find that everyone in the United States K — 12 system is 
trying to inform themselves about something, but — e.g., 
because of incompetent teachers — failing dismally. Would 
you feel that citizenship/social studies education was suc- 
ceeding? This is not a semantic issue. I suspect that, in some 
feeble sense, most people "try to ^understand" the use of 
bloody and destructive violence by political revolutionaries 
in this country. I think most of them (would say they) fail. I 
think that shows something about the gross inadequacies of 
citizehship/sociai studies education, not something about 
its success. They know nothing ^ of the philosophy of 
violence; they could identify none or at most one of the half- 
dozen powerful reasons for the use of violence; it does not 
even occur to theni that their own country was founded on 
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violepc^ anS has perpetrate(r=?2nd institutionalized violence 
to a massive extent. They are esamfHies of the failure of 
citiztsnshtp/sQcial studies education, and an evaluation 
should so identify them. * . ^ * - 

The next major failure of the Assessment's effort lies in in* 
. terpretation, and it really falls under tvyc subheadings: in- 
terpretations by staff and interpre^tations by consultants 
v/hose report was published by National Assessment. 

Here's an example of absurdly poor staff interpretation: 
"One indication that students do weigh alternatives 
rationally vvras seen In the group partipjpation exercises; 67- 
79% 'at all three school ages gave a reason foi' a partic- 
ular point of view at least once ducing the one half hour 
task/'" 

Giving a reason may be aimed at persuading others or 
rationalizing one's own decision; hence it is simply 
improper to take it as an indication of rational deciding. 
That error shows a very serious lack of understanding' of 
what rationality is, and that lack of understanding ^hows up 
frequently. 

What would 6e evidence of critical ability and of rational 
decision making? A case where prior prejudice vyon't give 
the right answer, where the answer must come by inference 
from the given facts of the case^ in short a new problem case. 
None occur under Goal G "Approach Civic Decisions 
Rationally." . 

There's, a pervasive overoptimistic bias in the interpreta- 
tions. Why should "^ne be inclined to think that young 
Americans' critical ability is anything less than ludicrous 
when a majority of 9-year-olds and a quarter of 13-year-ol3s 
think that a newspaper canH be wrongP^ I'hat's after six of 
seven years of schooling. 

Interpreting the global significance of the results was left 
to an advisory panel. I will indicate my interpretation of one 
small part and you'll see why I think the^ truly horrifying 
implications have not generally been recognized. Even with 
the data at hand, despite the many deficiencies already in- 
dic<ited, much more can be inferred than either staff or ad- 
vi^rs have recognized. The conclusions are not both precise 
and highly probable. But policy decisions, contrary- to the 
usual position, do not require these conditions. We operate 
off probabTilities and possibilities, when the risli of not doing 
, so is high; and in this area, that's surely the situation. 

Let's take the respect for freedom of speech.*^ It's often 
mentioned that 75% or more of the 13-year-olds thought 
that no one on radio or television should be allowed to say either that 
"Russia is better than the U.S." or that ''Some races of peo- 
ple are better .than others" or that "It is not necessary to 
believe in God." 

What isn't so often said (though National Assessment 
staff noted it) is that 94% drew tht line at one or more of these 
statements as a permissible media utteranee (i.e,, only 6% 
thought all were uttgrable). The 17-year-olds still show 
almost 80% refusing to allow all three, and the young adults 
still show 68% standing four-square against freedom of 
speech in these medium-controversy examples. When ssked why 
they thought these statements should be allowed, only two 
thirds of the most stalvygrt (adult) sample could think of 
freedom of speech or ideas, etc., as a justification. One 
should perhaps quote as the most significant statistic the 
76% of the adults who failed this simple test, treated as a 
simple recognition test of a well-known principle. Now how 
many of the remaining 24%, if on the bo2^rd of a broad- 
casting station, would actually stick to their verbal endorse- 
ment of this principle? The evid'cnce (from Hartshpj-ncLand 
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May on) suggests that it will be far fewer — perhaps only 
10% instead of 24%, perhaps only 2%. And are these exam- 
ples extreme tests? On the contrary. Suppose the third 
quote was not "It is not necessary to believe in God" but* 
"Belief in God is a sign of weak-mindedness and the source 
of most war and cruelty." Would we really have 20% lef^to 
count on? 

Remembering that huge gap between professed mdrat ' 
principles and actual practice, how should we feel about a 
test of professed tolerance of o|herv races under very mild 
stress as in A4 — "-being willing to have sorficone from 
another race be your dentist ordoctor, live next door to you, 
represent you in an elected office, sit at tht next table in a 
crowded igstauraht, ,stay in the samG moid or ho.tel" — 
when we find that 43% of all age levels draw the line at one or^ 
more of these possibilities. When it come5 to the day when 
the respondent's daughter actually wants to date inter- 
racially, one can have Uttlc confidence that half of that 43% 
will remain with us {and I'd. have to say that 10% would be a 
surprise). 

Is it not disastrous that less than a quarter of youitg adults 
(22%) could giv,e even one reason for and one reason against 
education deferntents folL.thc draft? 

, Now I would also'say thaf inost of the remaining questions 
are routine questions about routine bdhavior and 
knowledge, and the subjects performed routinely on them. 
One can draw little joy or sorrow from thqse other 
responses. But on the issues that test the capacity for crisis 
handling tn the citizenship domain — although the tests arc 
weak and the inferences from tcsi performances to real per- 
formance very shaky — the rcsulls I have quoted represent 
most of the questions asked (sinc^ there were very few), and 
surcjy they repr.es^nt significant Tea^ures of the answers. 

What did the Panel of Reviewer^' think" of these results?'" 
(Remember that their reactions represent the only evaluative 
global synthesis effort by NAEP)* 

By and lafge, they thought the nesuks were pretty en- 
couraging. A black panelist (Tobe Johnson) rightly com- 
plained about the "WASP" standards built into some ques* 
tioiis. Larry Metcalf saw the same point, and some other 
biases, and cautioned us not to blame or credit.the schools * 
for the results. 

But no one expressed horror at the plzt^n ignorance ahd 
prejudice revealed here, ar4dteeveral expressed gratification. 
Evaluation results sometimescall for horror, and these ones 
do. As to blaming the scliools, why not? There's no reason 
to think the schools couldn't change these results around tf 
they tried and there's every reason to think tfiey should try. 
No doubt faiT^sNes, communities and med^a are also to 
blame and woiiird also t'-esist the effort to change. That 
doesn't show it can't be done> and if it can be and should be 
and isn'tvthpn those who don't do it must share the blame. 
Communities can be changed by their schools; schools 
aren't petrified by communities in law though they ^ay be 
in fact. ' * 

6o i d sum up my reactions to the National Assessment 
effort as involving ^ratr weaknesses of design andinterpreta* 
tion, as well as great technical virtuosity in many dimen- 
sions. 



Comments on the Taylor Chapter 

Much of Taylor's excellent review is unexceptionable I 
will just mention one disagreement. 
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TayJorsaysj "Ths assessment movement is counter to the 
liumahizirzg movcmsnt In American education. It is 
promoting a closed rather than an open approach to cur- 
rfeulum. " I think this is a very serious misconception. To ex- 
pm schools to provide certain core learnings is not to in- 
hibit their room for all sorts of innovation. To expect stu^ 
dents to test well on understanding democracy is hardly in* 
foibiting humanizationi * » * . 



Conclusion ' 

I have tried to develop new perspectives on the evaluation 
of citizenship/social studies, partly by describing new 
models and part!y by criticizing the present ones. Lhope this 
will lead us towards mor^ ttx^/i// evaluation 9,vA more effecim 
education in this area. Nothing* in our national priorities is 
more important. ^ . 
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CHAPTERS ^ 

CRITIQUE OF NAEP OBJECTIVES AND PROCEDURES: 
CITIZENSHIP AND SOCIAL STUDIES 

A. Quy Larkins 
University of Georgia 



The purpose of this chapter h to help social studies 
educators determine the extent to which the findings of the 
National Assessnjtnt of Educational Progress (NAEP) are 
jDjrounded in sound practice. The following pages deal 
primarily with two topics — the objectives that NAEP used 
as a basis for developing test items (see Appendix A) and the 
procedures used to improve the quality of those it;ems. They 
also contain less-detailed comments about sarnpling and 
data analysis. 1 

Objectives | 

One of the difficult tasks faciiTig National AssoUmcnt was 
to decide which educational achievements ought to be as- 
sessed. NAEP's approach to this problem was to develop a 
set of objectives for each of the 10 learning arcak Test items, 
which were to have content validity for those objectives, 
were then ^written. The quality of test items in citizenship 
and social studies is dependent in part, therefore, on the 
quality of the objectives. That topic is discussed below in 
terms of two questions: 

1. Do the objectives meet NAEP criteria? 

2. Do they meet National Council for the Social 
Studies (NCSS) guidelines? 

NAEP Criteria 

It is reasonable io expect NAEP objectives in citizenship 
and social studies to meet criteria established by NAEP. 
Those criteria are: 

1. Specialists in the learning ar£a must consider 
the objectives authentic from the viewpoint of 
the discipline. Scientists must agree the science 
objectives are authentic; mathematicians must 
agree upon the authenticity of the mathematics 
objectives, etc. 

2. School people must recognize them as desirable 
goals . . . which schools are actively striving to 
achieve. 

3. Parents and others interested in education rtiust 
agree the objectives are important . . . 

Academic review. The first criterion may reflect a 
time-honored assumption that school subjects should be 
based on parent academic disciplines; but citizenship has no 
definitive parent discipline (^. A number of scholarly areas 
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are relevant to what ought to be taught under citizenship, 
but no discipline or group of disciplines stands in relation to 
citizenship the way that mathematics stands to school math. 

Another problem is that more than arty other Assessment 
area, citizenship is concerned with beliefs, attitudes and ac« 
tions that require ethic^ justification. Scholars in a dis- 
cipline cannot tell us, as a natural outgrowth of their 
academic training, what a student needs to know to be a 
good citizen in the ethical sense of the term good< Similar 
arguments could be ]rnade for social studies, especially to the 
extent that it includes citizenship education. 

Having commented on some limitations pf using scholars 
to judge the appropriateness of objectives, we now ask: How 
well was this criterion applied by NAEP? To what extent 
\^^ere scholars from the Various social sciences and history in- 
cluded in the development of social studies objectives? 
According to Frances S. Bcrdie, a NAEP representative: 
In the summer of 1965, 11 social scientists met for two 
and one-half days with members of the ETS staff to 
define the proper domain of an inquiry into the 
achievements of American education in this subject ^ 
area.^ ' 
Apparently, however, Berdie used the term "social scien- 
tist" loosely. Of th'fc 11 members of that committee, only 1 
was explicitly identified as an active scholar in those dis- 
ciplines that National Assessment claims are most relevant 
to social studies history, geography, economics, political 
science, anthropology, sociology and social psychology.^ At 
a later review conference, only 2 out of 19 participants were 
clearly identified with academic disciplines.^ Although Ber- 
die claimed that all 4 participants in a third review con- 
ference were social scirntists,^ they w?re each 'members of 
departments of education in various universities. <^ 

Similar comments seem to be justified concerning the par- 
ticipation^ of academic scholars in the development of 
citizenship objectives.^ 

Taught in schools, NAEP*s second criterion for objectives is 
that schools must be actively striving to achieve themi it is 
important to stress that this critei'ion has been consistently 
and repeatedly emphasized in the project *s publications." 
Therefore, it is surprising to find that the contracting agency 
responsible for developing citizenship objectives explicitly 
rejected that standard,^ and lhat the rejection is stated in 
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the midst of three separate claims by National Assessment 
that the criterion is important.*^ The obvious inconsistency 
U neither acknowledged nor explained. Nevertheless^ the 
majority of NAEP citizenship objectives appears to be 
among those that schools are striving to achieve, but some 
arc not.^V . ^ 

la it also true that some social studies objectives fall out-- 
side the usual domain of the schools? The answer appears to 
^ be "no," but only if those objectives arc considered one at a 
4imet If the total set of social studies objectives is taken as 
approximately descriptive of school social studies curriaila, \ 
children woiild receive an incredible dose of history and the 
social sciences. Far more content is implied by those objec- 
tives than schools can hope to teach. 

Lay review. Compared to usual practice, National 
Assessment made aiji earnest effort to, meet the criterion that 
parents and others must agree that objective^ arc important. 
Despite the" assumption that American public schools are 
answerable to citizens, assessments of educational achieve- 
ment generally do not involve laymen in determining what 
ought to be measured. In contrast, NAEP held a series of 
regional lay conferences to review objectives, and major revi- 
sions to the social studies objectives resuUed from those con- 
ferences.*^ 

Could NAEP have done better? Yes. Persons who at- 
tended the lay conferences were nominated by groups s^ich 
' as the American Federation ^of Labor, National PTA, the 
United States Chamber of Commerce and the National As- 
y sociation for the A'dvancement of Colored People. This 
selection procedure nearly guaranteed overrepresentation of 
middle-class viewpoints. 

Summary. How well did NAEP meet its own criteria for 
objectives? Tht\ans\ver is mixed: (1) Few people who are 
clearly identifiable as scholait in relevant academic dis- 
cipline:^ were listed as reviewers of social studies and 
citizenship objectives. (2) The contracting agency for the 
citizenship objectives rejected the criterion that they must 
be goals that schools are striving to achieve, and the social 
studies objectives imply far more content than^ schools 
seriously attempt to teach. (3) The criterion of lay review 
was taken seriously and achieved reasonably well, but 
greater diversity of social-class representation on lay panels ^ 
may be desirable. 
. The following section deals with the question: How well 
do NAEP objectives meet guidelines of the National Council 
for the Social Studies? 

NCSS Guidelines 

A position paper titled Social Studies Curriculum Guidelines 
.was published by the National Council for the Social 
Studies in the December 1971 issue of Social Education. The 
structure of most of what follows is taken from the rationale 
portion of that paper. It is divided into four subsections: 
Knowledge, Abilities, Valuing and Social Participation. The 
NCSS position on each of these topics is compared to the 
positions expressed in NAEP citizenship and social studies 
objectives. 

Knowledge, NCSS guidelines and NAEP ^social studies 
objectives differ on the manner in which social studies 
shov-ld be drawn from history and the social sciences. NCSS 
appears to define social studies primarily as citizenship 
- education. NAEP social studies objectives appear to define 
social studies primarily as condensed replicas of history and 
the social s<?iences.^* NCSS explicitly rejects that defini- 



tion J* Its position seems to be that those portions of the cur- 
riculum that are "selected' from histohr and the social 
sciences should be chosen for their relevance to the resolu- 
tion of social problems.** ' 

National Assessment citizenship objectives arc generally 
consistent with 'the NCSS position on the relation of 
academic disciplines to social studie^. Judicious selection 
from the disciplines, rather than an attempt to replicate 
. them, seems to have been the rule. Citizenship objectives I 
F, I G, Il^and V are typicarl^f those requiring that 
knowledge from the social sciences and history be applied to 
social problems;^' but NAEP also includes several objectives 
that are outside the scope^f the curriculum implied by the 
NCSS guidelines. For instance, NAEP citizenship objectives 
deal with such diverse topics as health and safety,^" family 
relations,»''social etiquette 2« and yocational education.^' 
Abilities. NCSS stresses divergent thinking, data processing 
aiid human-relations competencies.^^ One National Assess- 
ment social studies objective states that children should 
know that divergent thinking is useful to scientists, but it 
does not point to the need to foster divergent thinking 
among students.^^ Another objective, however, is consistent 
with the NCSS view on data-processing skills such as 
locating, organizaing and assessing data « and source 
material^^ Of the three human relations competencies cited 
by NCSS — sensitivity to others,* communication skills and 
ability to cope with conflict and authority — the first ando 
part of the ^ird are mentioned by National Assessment.-'' 

NAEP citizenship objectives attend to data-processing 
skills but are not as strong as social studi'^s objectives in that 
area. Citizenship objectives are strongei than social studies 
objectives in human-relations skills,^^ btit divergent-creative 
thmking is neglected, furthermore, despite statements 
about the importance of dissent,^'' which implies diversity of 
opinion, the general tone of the citizenship assessment 
presses for conformity in values and in the stances taken on 
political-ethical issues. 

Values. NCSS guidelines and both assessments agree that 
values should be dealt with in the school curriculum. 
Despite some ambiguity, NCSS opposes indoctrination of 
even, basic values such as those contained in the Bill of 
Rights.-^NAEP social studies objectives on values include 
the phrase "reasoned commitment" and, therefore, appear 
to be closer to the NCSS position than do the citizenship ob- 
jectives.^'^ Although one citizenship subobjectaves states that 
rights are not absolutes and that they frequently conflict 
with each other, ^" other objectives present values as un- 
qijjilified standards of proper behavior. In the majority of 
objectives that focus on values or on substantive social is- 
sues, the emphasis is on whether "Students take the "correct" 
stance, rather than on whether they make a rational 
choice. \* That emphasis is out of harmony with the spirit of 
the NCSS guidelines. . , 

Social participaiion. The central thflrust of the NCSS position 
on social participation is absent from NAEP social studies 
objectives. That thrust includes: **Extensive involvement by 
students of all ages in the activities of their community 
. . . .^^ Those activities range from political campaigns to 
making important instructional and curriculum decisions in 
the school. * 

Compared to NAEP social studies objectives, far more at- 
tention is given to social participation in the citizenship as- 
sessment. It is difficult to find a citizenship objective that 
does not include involvement by students in social problems. 



They arc based, however, on the assumption that correct 
social participation can be identified and ought to be ex- 
pccted.3^ In contrast, the NCSS position is more consistent 
with assessing the reasoning behind the positions taken by 
students, rather than assessing whether those positions fit 
majority expectations. 

Postscript to ObiQctives 

The first half of this chapter has examined the question: 
How well, do NAEP objectives meet NAEP criteria and 
NCSS guidelines? Agaiathe answer is mixed. Some NAEP 
•criteria were met better than others, and some objec'Hvs 
met NCSS guidelines better than others. In the final 
analysis, however, the critical question is: How welPdo 
NAEP objecti^ves fit your assumptions about the proper con- 
tent for social stiidies? It is this writer's opinioh that those 
objectives will completiily satisfy few people. But neither are 
the NCSS guidelines likely to completely satisfy most \ 
members of our profession. However, teachers who read and 
compate those objectives and guidelines will find many im- 
portant statement!|with which they agree and many others 
that will stimulate them to clarify their opinions concerning 
appropriate goals for social studies. 

Exercise Development 
Introduction 

Perhaps the most crucial task facing National Assessment 
wa3 the development of valid measures of achievement. The 
criteria and procedures used by the project to develop 
citizenship and social studies exercises are examined in the 
present section of this chapter. Whether those exercises are 
valid is . discussed elsewhere in this report. Among*the 
criteria used by NAEP to judge the quality of their assess- 
ment instruments were thoae having to * do with ofFen- 
siveness, content validity, clarity and difficulty level. Among 
the procedures usedT^)' iipprovc exercises were: <1) reviews 
by laymen, subject Anatter specialists, technical advi^rs 
and the United States OITice of Education (USOE), and (2) 
various field trials. ' 

Lay Reviews ' ^ 

One of the innovative features of National Assessment is 
that laymen were asked to review exercises to help insure 
that the<^ssessment instruments would be acceptable to the 
general public. Althpugh this is a useful and important way 
to-involv€4a^uncn4^ it may have serious consequences for the 
content validity of exerciscis. For instance, it might not be 
possible to test some parts of a legitimate social studies or 
citizenship topic without getting into controversial 
problems. In those casesi^ the ofFensiveness reviews might 
cause exercises to be so modified that a decrease in ofFen- 
siveness is accompanied by a decrease in validity. The 
potential danger to the validity of citizenship and social 
studies exercises is illustrated by the following selection of a 
few of the topics which the lay panels found offensive: 

, References to, specific minority groups should be 
eliminated whi^ever possible ... 

Any reference ^o . . , the FBI, the President, Com- 
munism aad specific prganizations such as the Ku 
Klux Klan and labor unions might make an exer- 
cise offensiv^ unless extreme care was used in the 
wording .... 



' Exercises which show national 4ieroes in an un- 
complimentary f^ishion though factually accurate 
are offensive. 

' Exercises which might be interpreted as putting 
the police or other authorities in an unfavorable 
light are offensive.^^ ' 

The above examples were chosen because they seem to 
have more j)otential that^ others for reducing content 
validity. Examples could ^Iso have been selected that might 
improve validity. For instance, the lay panels objected to 
questions that violated the privacy of families and that ex- 
pressed ethnocentric views.'^ 

It is impossible to judge the extent to which the lay 
reviews affected content validity But the potential impact 
was considerable. According to Finley and Berdie, 
'^Citizenship exercises were reviewea and revised and re- 
reviewed so often that no figure [on the number revised as a 

result of the Iay4:bnicrcnces] is m(ianingful Each of 

the five lay conferences exaniined citizenship exerciSes, and * 
three! of tlic five examined social studies. Only literature 
came close to receiving as much attention as the two areas 
most relevant to social studies educators. Our profession, 
therefore, should be particularly interested in the possible 
impact of the lay panels on content validity. 

Other reviews that were.relevantto the criterion c*" ofFen- 
siveness include: (1) a lay conference that focused on 
whether exercises were trivial; (2) a review by the NAEP j 
Technical Advisory Committee and (3) a review by USOE. 
Results of the USOE review illustrate that the criterion of ^ 
ofFensiveness can have special impact on our area of the cur- 
,riculum. Three of 4 exercises that were dropped, and all U 
•exercises that were modified, were in citizenship. 

Subject-Matter Reviews 

Following the lay reviews, exercises were sent to subject- 
matter specialists nominated by professional organizations 
such as the National Council for the Social Studies. 
Althpugh this, mailed review proved helpful in identifying 
problems with content validity; as few as two reviewers were 
used for each learning a>ea It was impossible for reviewers 
t|^ determine whether the larger content of an assessed 
learning area was properly sampled because each reviewer 
received only those objectives upon which the exercises Ije 
critiqued were based. Therefore, National. Assessment 
decided to hold a series of subject-matter review con- 
ferences.^^ 

At least four subject-matter conferences were held to ^ 
review exercises for the social studies or citizenship assess- 
n^ents. Conferences were also held to produce additional ex- 
ercises for 13-ycar-olds and for young people from minority 
groups. - ' " x:> 

An impression left by this series of conferences is that 
NAEP was determined to produce acceptable^ appropriate 
and valid exercises. When the mailed reviews were not ade- 
quate, conferences were held to allow reviewers to meet face . 
to face. When early conferences indicated a need for greater 
attention to assessing the achievement of ^minority group 
students, additional writing and review conferences were • 
held. After each review, questionable exercises were* 
returned to the contractors for modification. In some casfes, 
new exercises were produced, which were also reviewed by * 
subject-matter specialists. In terms of sheer number of 
reviews and revisions, it is diffifcult to imagine how the pro- 
ject could have given greater attention to face validity 
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without violating their budget and production schedule. 
However, National Assessment may have been able to 
improve the reviews by using a more systematic and 
thorough within jconfere nee, procedure such as that 
described by Hunkins.^^ * 

Field Studies 

Thti content validity of exercises was determined 
primarily tlmugh the subject-matter reviews described 
above. Scverat^^^^d studies, however, focused on criteria 
that arc relevant to the quality of test items, such as clarity 
and difficulty level. 

The A^essment specified that each exercise must be writ- 
ten to meet one of three difficulty levels — very easy, 
moderately difficult or very difficult. An easy item is one 
that can be correctly answered by approximately 90 percent 
of the students*"'' One study attempted to determine 
whether "easy" items met. the 90 percent criterion. The 
answer was "no." Exercise writers missed the mark by a 
wide margin.*^ Therefore, NAEP took special pains to 
produce additional easy exercises. Success in meeting the 
other difficulty levels was not deteri^ined. 

Because high difTiculty resulting from lack of clarity in 
any item was not desirable^ three feasibility studies u&ed in- 
terviews to spot problems in understanding test instructions^ 
format, .vocabulary and vague or ambiguous terrhs.*'' The 
impor^nce of determining whether students understand 
test questions is dramatized by one of tht results of a study 
that used low-achieving children: "At the 9-year-old level, 
the three students were such poor readenf that each exercise 
had to be read aloud before they were able to answer."*'^ 

Two studies that focused in part on whether changes in 
format affect the difficulty of exercises vverc characterized by 
high internal validity.'*'^ The major conclusions of these 
studies were: open-ended questions tend to be more difRcult 
than multiple-choice; the difficulty of multiple-choice exer- 
cises can be manipulated by changing the distractors>«nd 
including *4 doo^tjcnow** tends to reduce the number cor- 
rect response's to multiple-choice^ items. Since NAEP uses 
both multiple-choice and open-ended exercises, these results 
mean that caution should bemused when comparing the 
result to different exercises. What may appear to be a dif- 
ference in knowledge^ simply be a difference in the dif- 
• Acuity of the item formats. . " ♦ 

After exercises had been extensively revised as a result of 
the lay reviews, subject-matter reviews and initial field 
studies. Anal tryouts were held prior to the selection of items 
for inclusion in the actual assessment.''^' Exercises that were 
to be individually administered were tried out by interview- 
ing six persons per item. Exercises that were to be group- 
administered were tried out by testing classrodm4ize sets pT 
students. 

^*^!Thc use' of interviews in the tryout of individually ad- 
ministered exercises allowed for direct assessment of the 
clarity and difficulty of test items, but an indirect approach 
was usec^ with the gr6up-adm(nistered exercises. The class- 
room teacher and a representative of the contracting 
agency responsible for the tryouts each completed an obser- 

"^ation forrfi, which contained categories for such inap- 
propriate student behaviors as apparent inattention or 
boredom. 'The intervie\y procedure seems to be superior for 

' detecting problems that are relevant to the validity of exer- 



Summary 

This author is impressed ^<itl^ National Assessment's 
emphasis on producing exercises that students can under- 
stand. The number oT reviews, revisions, field studies 
and additional revisions indicates that this criterion was 
taken seriously. Nevertheless, NAEP seems to be pper^ 
ating on two questionable assumptions about how to im- 
prove exercise clarity. 0 

The first assumption Is that items that are clear to low- 
achieving students will also be clear to more able ones.^' 
J. The opposite may be true, for instance, in cases -where 
^ bright students are aware of the multiple meanings of am- 
biguous t^rms.' 

A second questionable assumption is that experts caiy 
make adequatejudgments about whether an exercise will be 
understandable to children. The exi stenc e of field trials docs 
^not negate this assumption* Although National Assessment 
conducted several studies that were relevant to clarity, no 
fielci trial contained all of the following features: (1) inter- 
views to uncover communication difficulties, (2) inclusions 
of all exercises ihs^/T might be used in the firial assessment 
and (3) adequate samples of respondents of vari6u$ ages; 
abilities and backgrounds. . 
. Despite the above criticisms, clarity is an important 
Criterion, which NAEP tried diligently to meet. < 

In this writer *s opinion, the criterion of three^evels of dif- 
ficulty is not as important as clarity. Even so, the criterion is 
a good<)nc, which helped the Assessment avoid the^narrow 
vision of assessing only a mid-range of achieveipnents. The 
concern for difficulty levels also had an ufianticipated 
benefit: failure to meet the 90 percent cciterion forced a 
closer examination of the clarity of exercises. 

- Reliability 

An unusual feature of NAEP exercises is that standard es- 
^timates of reliabilky are inappropriate. Readers who are ac- 
^ customed to seeing reliability reported as coefficients, such 
'ts .84 or .91, may bepuzzled by the lackofisuch statements 
^ in Assessment reports. Types of reliability, however, that 
yield coefficients cannot be computed for single-item tests. 
An alternative is to report standard errors. Ahhough this 
approach may be^unfamiliar to many readers, they can de- 
pend on NAEP reports to be cautious in the narfative 
descriptions of findings. The language used to discuss 
findings is carefully chosen to reflect the amount of trust 
that can be placed on their reliability. 

Sampling anc| Data Analysis 

In educationa/research, samples are frequently composed 
of local volunteers such as a few social studies classes in - 
schools that are willing to cooperate. In contrast^ National 
Assessment uses careful and thorough procedures that com- 
bine randomization with multistage cluster sampling, 
Readers of NAEP reports can be reasonably certain that the 
sample selected for each group in the assessment, such as 9-, 
year-olds, is similar to the national population for that ^ 
group. . 

Readers can also be reasonably certain that the 
. procedures used to analyze A^essmerit data are ap- 
propriate. The technical competence of the NAEP staff and 
advisory committees is impressive. The cautious language 
used to report the technical aspects of the Assessment 
leaves the impression that they are aware of the proper ap- * 
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plications and limitations o£the procedures used to analyze 
assessment data. > ^ 

Conclusion 

Despite the several criticisms in this article, it is the 
overall impression of this writer that NA£P used reasonable 



procedures. The technical aspects of the project appear to 
be sound. Most of the innovations, such as using laymeiito 
review exerciseSi appear to be useful. Although a project as 
missive as National Assessment is bound to run into dif- 
ficulties, even some failures^" there are few research efforts in 
education of this scope and quality » ? . 
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CHAPTER 6 

VALIDITY OF SOCIAL STUDIES AND 
CITIZENSHIP EXERCISES 

Francis P. Hunkins 
University of Washington 



The prime purpose of Task 3 was to determine the coiv* 
tent validity of the sockl studies and citizenship ekercisv^s 
developed by the National Assessment of E^uc^tioudit 
Progress (NAEP). A secondary purpose of the investigation 
was toindicate if the exercises as developed cquld be utilized 
as models by teachers in developing their own evaluation in- 
struments. Related aspects were considered as well by this 
investigator: the cognitive and aficctivc levels of the cxfer- 
cises» the format of she exercises, the manner in which the 
exercises were administered and the age levels to which the 
exercises were geared. » 

Content validity is the most crucial eriterion of any test 
exercise, for it appraises whether the exercise assesses what 
it is assumed to be measuring. Ointenf validity centers on 
the representativeness or sampling adequacy of the content 
of the items. Such validity is basically concerned with the 
question, "Is the substance or content of the item depictive 
of the content or the universe of content being measured?"* 
Specifically, ^he question relating to the NAEP exercises is 
"Arc these exercises representative of the objectives as 
developed by the National Assessment of Educational 
Progress?" Are the exercises dding what they are purported 
to do in relation to the objectives?" ^ • 

Social Studies, Citizenship -~ 
Goals and Objectives of 
National Assessment 

Since content validity is judged in relation to goals and 
objectives delineated, the reader is provided with an ab- • 
brcviatcd listing of the objective^ in order to interpret this 
chapter more completely as well as to formulate some judg» 
ments relating to the value of the ovefrall thrust of assess-^ 
ment in this area. 

Social Studies Qbiectlves 

Social studies is that area of the school curriculum 
that seeks to qommuiiicate about man in society. It is 
a shorthand term for such subjects as history, 
geography, economics, political sciences, 
anthropology, sociology and social psychology. 

I. ' Have curiosity about human af- 
fairs, 



in.' 



IV. 



V. 



II. Use analytic-scientific procedures 

effectively. 

Arc sensitive to creativeMntuitive 
methods cf explaining the human 
condition. 

Have knowledge relevant *to the 
major ideas and concerns of locial 
scientists. 

Have a reasoned commitment to 
the values that sustain a free 
society.* ' ■ 

Each of the above objectives had several subdivisions with 
descriptions as to what knowledge and/or behaviors should 
be possessed or evidenced by individuals at various age 
levels. Some of these objectives had lengthy discussions of 
subundcrstandings or subbehaviors. In determining the 
content validity for each item, the invc-stigator an^ three 
judges scrutinized these descriptions under the major objec- 
tives. 

Citizenship ObUctivBs 

The National Assessment of Educational Progress did not 
cerate a precise definition of citizenship; however, its in-r 
terpretations ^of citizenship are evident through the stated 
objectives. 

Show concern for the welfare and 
dignity of others. 

Support the rights and freedoms of 
all individuals. / 
Help maintain law and order. 
Know the main structure and 
functions of our goven;imept. 
Seek community improvement 
through active, democratic par- 
ticipation. 

Understand problems of inter- 
national relations. 
Support rationality in communica- 
tion, thought and action on social 
problems. « 
Take responsibility for own per- 
sonal development and obliga- 
tions. 



I. 

II. 

nr. 

IV. 



VI. 



VII. 



VIIL 
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IX. Help and respect their own* 
families (ages 9, i3, 17). 

X. Nurture the develjjpment of their 
children as futurc^itizens (adults). 

' Specific. Nature of the Task 

As previously indicated, the Task 3 investigator was to 
Review independently the NAEP social studies and 
citizenship exercises in order to inform NAEP, school per- 
sonnel and the general public as to the extent to which they 
can place confidence in the exercises. ' Basically, the in- 
vestigator analyzed exercises having one of four possible for- 
mats: completion, writing, multiple-choice and combjna* 
tion. The completion exercises required the student to fill in ' 
. a blank or blanks or to provide a short answer following 
some stimulus or to arrange alternatives in correct locations. 
The writing responses required .a student to vknrite a 
sentcnce(s) or paragraph(s) or list responses.^^e multiple- 
choice exercises asked the individuaj to select from twb or 
mbre alternatives^ and the combination question requested 
the individual to select from one or moxfi alternatives and 
then to respond by completion or writing. 

Procedure for Processing the Exercises 

The investigator was .assisted by three panel members in 
pro( essing* the exercises to estimate whether tljey measured 
the objectives intended^and to record descriptor information 
• ^ on each exercise dealing with age level, format, manner ad- 
ministered and tjieMike. Data were recorded on a format 
sheet identical to the following one: 



Format for Recording Exercise Data 



Learning area: 
Item: 

Item forihat: 

Individual or group administered: 
Age: 

Objective: 

Cognitive-affe<!tive levels 
Content validity: 
Context validity: 
Model for teacher: 
Age appr^opriatehess; 
Other: , 



The sequential instructions followed by the investigator 
- and his assistants for processing both, the social studies and 
citizenship exerdses were identical : 
^ 1. Read each exercise and record descriptor information. 

2. Make judgments as to whether the exercises had con- 
tent validity, context validity, exemplary qualities for 
teacher use and age appropriateness. 

3. Read appended responses to each exercise if present. 

4. As a safeguard against misjudgment reconsider the 
content-validity question with regard to how in- 
dividuals actually responded. 

Judges* responding independently to all exercises, in- 
itially agreed on ^the content validity and exemplary 
qualities of a surprising 95% of the exercises. Then judges 
and the major investigator as a group discussed their judg- 



mAts of the content validity and exemplary qualities as well 
as context validity, cognitive/aflcctive leyels and age ap- 
propriateness. In cases of disagreement on content validity 
and exemplary qualities reasons for differences were con- 
templated. The group hadMittle difiiculty^ reaching com- 
plete agreement on "Toritent validity- and cx^plary 
qualities. The investigator put aside for additional delibera- 
tion those exeVcises^where consensus was not achieved on 
other categories of secondary interest to this report and later 
made the final judgments on such matters. 

'The Results 

\ • 

Results are reported for released and unreleased social* 
studies exercises and for released and unreleased citizenship 
exercises. (Readers may be interested to know that the 
panel found the unreleased. exercises in both social studies 
and citizenship^ similar to the released exercises.) 

Social Studies Exercises: 
Peieased and Unreleased 

Table 1 summarizes the data relating to all soci^ studies 
exercises* both released and unreleased. The table records 
. information relating to the total number of itcms^for each 
major objective as well as the number of exercises for each 
subobjective. Since this chapter lists only the major objec- 
tives, individuals- wishing information about the specific 
subobjcctives should refer to Appendix Ar — 

Perhaps the most significant point is that of the 194 exer- 
cises prepared for social studies, 85 percent (164) of them 
were considered valid by the investigator and the panel 
members. This should enable us to have somcf degree of faith 
in the exercises ^s truthful in measuring what they state they 
are measuring.j 

Just looking at the overall percentage of exercises can be 
misleading. Not all objectives are represented by equal 
numbers of exercises, and also the exercise^ emphasize dif- 
ferent cognitive and affective levels. 

Objectives 11 and IV had significantly more exercises 
prepared — 51- and 83 exercises, respectively. Whether this 
suggests a hierarchy of^importance regarding the objectives 
remains to be seen. Perhaps this loading is due to the objec- 
tives themselves: Objective II rjslates to whether individuals 
use analytic-scientific" procedures. With the emphasis on 
p^cess today, perhaps this distribution of exercises refiects 
current curriculum status. However, this inference is 
somewhat suspect, for if process were the emphasis, the ex- 
ercises would not cluster at the lower cognitive levels. 
• Perhaps the number of exercises is related to the number of 
subdivisions in the objectives; for example. Objective IV has 
six subdivisions. Or perhaps the number of exercises reflects 
simply the technical difllculties of creating feasible exercises 
for some subobjcctives. . Moreover, all exercises were 
reviewed by panels of lay persons, subject-matter specialists 
and the United States Office of Educaiton (USOE). Since 
social studies — and citizenship — are often controversial, 
some exercises were likely to be ruled out in this review 
process. But if NAEP has stated that social studies learning 
should be related to five major objectives, then all dimen- 
sions of these objectives sj;iould be presented adequately by 
exercises. That no exercises exist for Objectives IIB^ IIF, 
MIC and VD and that only one exercise exists for Objectives 
niA and IVA makes it impossible to assess whether in- 
dividuals are demonstrating behaviors couched in thes? ob- 
jectives. National Assessment needs either to eliminate these 
subo^ectives or to create exercises for them. 
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citizenship Exercises: 
Released and Unreleased 

Table 2 reports a summary of data relating to all of the 
citis^enship exercises. Skimming the table allows one to stc 
which objectives were stressed, which were ignored 
regarding exercise representation and the content validity of 
these exercises. As is true with the social studies exercises, 
the citizienship exercises, both released and unreleased, 
favor certain objectives. Objective I, **Show concern for tKc 
welfare arid dignity of others," had a total of 22 exercises 
while Objective IV had 46 exercises. The emphasis on Ob- 
jective IV, which deals with the main structure^and function 
of our government, might suggest that citizenship as con*, 
ceived by the exercise developers, is primarily political. 
However, the overall objectives do suggest a* broader in- ' 
terpTetation of citizenship. Moreover, several citizenship 
subobjectives have no exercises at all. 

It would be important to know whe^ther the number of ex- 
ercises per objective represents a conscious plan or is the 
result of just how the exercises happened to fall as developed 
and approved. This disproportionate weighting of some ob- 
jectives is a problem that needs attention. 

Not only did the investigator and the panel mer ibcrs find 
these gaps in exercise existence, but they judged c nly 93 (or 
6h percent) 6f the total 152 exercises developed as having 
content validity. One needs to exercise extreme caution in 
assessing the level of citizenship functioning of individuals 
when 39 percent of the exercises are not valid. Even fewer 
were considered exemplary — only 55 percent. 



Final Comments 

As a whole, the panel concurred with the investigator that 
the social studies exercises as a total group, both released 
and unreleased, were superior to the citizenship exercises, 
both released and unreleased; a greater percentage (85 
percent) of social studies exercises possessed content validity 
as opposed ^to only 61 percent of the citizf^ship exercises. 
Also, a greater number of social studies exercises were 
deemed exemplary. 

Interpreting the results of National Assessment is tied up 
with whether one favors or opposes any national assessment. 
However/if we assfume that the reader is in favor of national 
assessment, and if we assume that the majority of the exer- 
cises are valid, it is still dilHcult to determine precisely 
whether the 'schools are tp be praised t>r blamed for the cur- 
rent levels of understanding and functioning of individuals 
in the area of socia^studies and citizenship. Certainly, the 
schools are not the only institutions in society that ecfucate. 
Therefore, National Assessment can telLus what the levels of 
understanding, skills, attitudes and so on are^of various 
groups in our population, but it cannot tell us that the$e'" 
levels are entirely the results of good,or poor schooling; But * 
schools do need feedback even if of a general nature. The 
Task 3 investigation points up some needed adjustments in. 
certain exercises and objectives to afTord educators even 
more reliable input for their decision making about what 
schools themselves need to do to enable individuals to 
become truly effective persons, citizens for the present ,3nd 
future. 



NOTES 



1. Fred V. KcrWxvicr, Foundations of Behavioral Research (New York: Holt, Rinehart and Winston, 1965), p. 446. 

2. Sec Ahpendix D, Reference 14, pp. 9-27. 
3., Sec Appendix D, Reference 1, pp. 7-39. 

4. Cognitive and affective levels were defined in two sources. Benjamin S. Bloom, A Taxonomy q/ Educational Objec- 
tivfs: The Cognitive Domain (New York: David McKay, 1956), and David Krathwol, A Taxonomy of Educational ObjeC' 
iivfs: The Affective Domain (New York:' David McKay, 1956). 
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CHAPTER 7 




A RATINOW SOCIAL STUDIES EXERCISES 
SOCIAL STUDIES EDUCATORS 



June Chapin 
College of Notre Dame 
Belmont, California 



When it comes to interpreting the findings of assessments 
in citizenship and social studies, thoughtful people must ask 
whether actual performance levels arc adequate. The 
National Assessment of Educational Progress (NAEP) aims 
to report; it does not aim to congratulate or deplore. Judg- 
ments are left to dtfiers. * 

One task of the National Coucil for the Social Studies 
(NCSS) review, then, was tackling the hard question of 
satisfactory performance levels ort social studies exercises. 
(Unfortunately, neither time nor resources were available 
for asking like questions about citizenship exercises.) The 
results* when all was said and done, came from exercises. 
Although social studies exercises were» indeed^ related to 
objectives* every exercise was also expected to have some 
significance of its own, to stand on its own. Consequently, 
some examination of the worth of the exercises therruelves 
seenr^ed needed. Moreover; since the exercises were written, 
NCSS had published its Social Studies Curriculum Guiddines} It 
seemed sensible to ask whether the Exercises were in line ' 
with these guidelines. 

A panel of^ine social studies educators was accordingly 
selected. (Their names along with members of other panels 
appear^ elpcwherc in this report. )ln addition, the author of 
this chapter was designated chairperson with responsibility 
for designing rating sheets and processing data. Members of 
the panel came from different geographic areas. They were 
comparatively young — seven were under 45 and three un- 
der 30 years of age. The male-female ratio was about equal. 
About a third were members of minority groups, important 
in the light of questions about the suitability of exercises for 
such groups^ Professional backgrounds were varied, but 
social studies consultants and higher^education faculty were 
decidedly better represented than classroom teachers. 

A training session wai: iield using the NCSS guidelines 
and rating forms for each major matter to be examined. 
Members of the panel practiced on exercises from the 
citizenship assessment for independent ratings and for 
group discussion to clarify criteria. After the training session 
each rater proceeded independently. Exercises were iden- 
tified only by code number, not by objective. Panel members 
agreed more often than not, but at times spread their 
responses across all the available categories. When disagree- 
ments in ratings did occur, they were reported out, not 



buried. Differences in ratings came frcim ambiguity in the 
NCSS curriculum guidelines, varying interpretations of the 
meaning of the exercises and outright differences in the 
judgments of the raters. , \ 

Since each panel member made literally over^ 1,000 
ratings — many exercises had subparts — the ratings were 
processed by computer. ^Imost 3,000 pages were turned 
out . Although the full NCSS report includes complete data, 
space limitation^ require that summaries only dnd some too 
few illustrations of ratings of ''realistically satisfactory" per* 
formanc'e be included in this report. 

Compatibility with NCSS Currlfculum 
Guidelines 

National Assessment states: ^'Social Studies is that.area of 
the school curriculum that seeks to communicate about man 
in society. It is a shorthand term for such subjects as history, 
geography, economics, political science, anthropology, 
sociology and social psychology. . . At this point readers 
may want to refer to the social studies objectives listed :r , 
Appendix A. NAEP obj< dives, at least, appear to define the 
social studies as mainly a mini^ure version of- the social 
sciences. 

The NCSS Social Studies Curriculum Guidelines offer a 
somewhat diflercnt perspective. The guidelines* see the 
social studies as essentially* citizenshtfi education — 
although their definition differs some from that of NAEP's 
citizensliip objectives — and emphasizes social problems. 
The guidelines identify^four^ integrally related conriponents 
of social studies education: (1) knowledge; (2) abilities in 
thinking, human relations and obtaining information; (3) 
valuing; and (4) social participation. These guidelines are to 
be viewed as both a stimulus and a guide to evaluate existing 
social studies curriculum and to work for irtiprovement. 
They may be thought of more in terms of what ought to be, 
according to' the professional organization, than what ac- 
tually exists in schools. The curriculum guidelines may 
reflect more of the spirit of the times and more of the cutting 
edge. Nonetheless, considerable overlap appears between 
them and Assessment objectives. 

One task of the panel wasi to point out more precisely the 
relationship between social studies exercises and the NCSS 
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guidelines. The panel worked with the following questions 
Into which of the wain components of the NCSS guidelines do you 
think this exercise best fits? ( 1) knowledge, ( 2) abilities, (3) valuing, 
{ i) social participation? 

The panel allotted (1) 46% of the exercises to knov/ledge, 
(2) 31% to abilities, (3) 19% to valuing and (4) 2% to social 
participation (and 2% lo "no response^'*). 

Strikihg is the very small percentage placed in the "social 
participation" category. NAEP has separated assessment of 
citizenship and social studies. Had the panel rated the 
citizenship exercises, this category may well have increased. 
.An examination of released citizenship exercises appears to 
support this claim.^ However, the panel's ratings do give 
some indication of National Assessment's view of the social 
studies^; social participation is not stressed. 

According to the panel's rating, "abilities" plus 
*' valuing" came to about half of the social studies exercises, 
and "knowledge" the other MU. Many may be pleased that 
NAEP social studies exercises, in the judgment of the panel, 
give one fifth of their total to "valuing," Animportant ques- 
tion is\v1iether the "ki\owledge" category is too high. Still, 
in practi'Tc. the testing programs of most schools probably 
devote a much higher percentage .of test items to 
"kn9w^edge."^any teachers and much of the public may 
be satisfied with'^the degree of attention given to the 
*knowledge" category in the exercises. 

Desirability of the Exercises 

NAEP exercises might be balanced among the four com- 
ponents discussed above and $till be considered of little 
value or hardly in keeping with guidelines recommenda- 
tions. It seemed important that the panel give a formal and 
careful estimate of the general worth of each exercise on its 
own. As another task, 'then, the panel members' rated by 
these directions: ^*Using the NCSS guidelines as criteria^ estimate 
the general desirability of the exercise: (J) little^ (2) some, (3) 
moderate, (4) high or (5) very great." 

The panel rated the exeAises as shown in Table 3. 



TABLE 3. Summarv of NA£P Social Studies 
Exercises Classifiea by GeneYal Desirability 



Desir 



Very great 

High 

Modeiate 

Some 

Little. 

No res|}onse 



Total 



Percentage of 
Exercises 

26% • 

40 

21 

6 

3 

4 

100% 



Nutnbei;of 
Ojjservatsons 

879 . 
1,356 
722 
194 
97 
118 



3,366 



If "desirability" is defined as including the first three 
categories, "very great," "high" and "moderate," then ap- 
proximately 87% of tJ«f exercises, by the panel's criteria of 
NCSS guidelines, are desirably. If the first two categories, 
"very great" and "high»" are used, the panel judged two 
thirds desirable. Fewer than 10% of the social studies exer- 
cises were rated as of "some" or "little" worth, but NAEP 
might do well to look carefully at these exercises. (Readers 
may. fiijd-it helpful to compare these ratings with those of 



Francis Hunkins' Task 11 panel in Chapt^^r 6 of this report. 
That panel judged validity for "NAEP objectives without 
reference to NCSS guidelines,) . 

Realistically Satisfactory Psrformane© 

Schools have long used standardize 'tests. Ordinarily 
such tests were constructed to sort omKe most able from' 
the least able students — that is, the midscore was to be 
roughly "half correct" to allow the most and least able to 
spread themselves u^ and down the distribution scale. Since 
some benchmarks by which to interpret were needed, mean 
or median scores were obtained. Lack of better benchmarks 
than these meant that ^adequate performance was ^in- 
terpreted as the average of the population on whom the test 
was normed: it was "good" to.be above the norm and **bad" 
to be below. 

National Assessment, however, did not aim-to distinguish 
the most or least able individuals but to find out wh^t 
proportion-of young people could or could not do important 
tasks. NAEP'» therefore, movpd away* from norm- referenced 
tests and over to performance tests. Ailthough NAEiP has 
presented "national percentafges of success" as a way of 
summarizing findings, these are not norms in the older 
sense. (The temptation to regard them so "will be strong,) 
Moreover, 50% success can not necessarily be consicfered 
"adequate," nor 75% "good." ^ 

For example. 17-year-olds were offered the following 
exercise. 

Below is a discussion that was held in 1966. As you 
read it, try to decide what the two speakers 
primarily disagree about. 

Speaker I: The United States should fight a. 

limited way in Vietnam while seeking 
a negotiated settlement. Winning of 
;he war'in itself won't dd any good. 
The U.nited States mus\ aim instead at 
seeing that the South Vietnamese have 
improved education, democratic 
government, security of life, an^l then * 
deal with poVerty and the lack of o 
medical care. Financial aid, advice ' 
, and, technological know-how are what 
are really needed, but .... 
Speaker 11: Improving living conditions is a good *^ 
{iidea, but our primary job is fighting. 
The United States can't pei^mit itself 
to be pushed out of an area where it is 
* committed. If We withdraw, we would 
be telling that p^rt of the world 
threatened by Communist aggression 
that we either cannot or will not rhain- 
tain our position. All that really mat- 
ters is our power position in inter- 
national affairs. * ^ 
What do the two speaker;^ primarily disagree 
about? \ 

□ What power and poverty mean in international 
affairs. 

□ Whether the United States is actually capable 
of controllinc: .South Vietnam by force. 

□ The extcnt^to which the United* States shotcld 
be involved in Vietnam and the fnotives for its 
iiivolvement. 
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□ Whether Communist aggression in Vietnam is 
worse than a lowered standard of living in the 
United States. 

O I don*t know. 

Seventy-Rve percent of the H-year-olds were successful/ 
Is this "good" performance? ('Die panel thought "yes" and 
{luidj^ed that 61-80% should be able to perform successfully.) 
*rhirteen*year-oIds were asked: 

A, Do you think teenage students should help 
decide what courses will be offered in their 
£chooS system? \ 

~-Ye3 Nq.^-^ — ^ ^Undecided * , 

II. ^Please give a reason for any answer you 
selected. 

0 " NAEP reports that 64% cf the 13-year-oIds gave accep- 
table reasons (or their choice in part A.^ Js this "good" per- 
formance? (The panel thoughl "yes" and judged that 61- 
i?0<^S ^iiould^ be able to perform successfully.) 

^!o?eover, National Assessment exercises were written at 
very rQi^hlylhrci-drfTacutty levels; on some exercises 90% 
were expected to respond correctly, on others 50% and on 
others only 10^. While considerable discussion could be 
qiv^?n here to the issues inherent in writing exercis^with ex- 
pectations' of S00% success, or "mastery," suffice it to say 
here that NAEP, for good reason, decided against exercises 
written with such expectations. 

The problems of giving meaning to performance levels 
and of clarifying ideas about what is significant to assess 
seem clearer when attempts are made to say what satisfac- 
tory performance levels are. The panel's judgments arc 
simply those of nine people, although competent people. 
Their ratings. However^ may give some guidance. 

The NCS.S panel was asked this question: "Realistically 
what level of performance nationally for the age leveLbeing- 
considered would satisfy you for this exercise? (1) less than 
• correct/a)^30^0%, (J) 41-60%» (4) 61-80% or (5) more 

_than 80S^o?" Notice that the panel's purpose was not to guess 
or predict an actual level of performance but to consider 
tvhat a satisfactory level should be. Notice also that the 
panel aimed to be "realistic." It is easy to hope that 
everyone will be able to do "everything." In present educa- 
tion and social conditions and with all the variations among 
^ younq; people, defining "satisfactory" as "everyone^ should 
do it" is to dodge the problem. 



Although a summary of rating perhaps obscures more 
than it clarifies, a summary is made in Table 4. 



TABLE 4. Summary of NAtlP Soda! Studies 
^ Exercises Classified by National Level of 
Realistically Satisfactory Performance 



Realistically Satid'actory Percent of 


Number of 


Performance Levels 


Exercises 


Observations 


Less than 20% 


2% 


79 


Between 20 and 40% 


3 


186 


Between 41 and 60% 


13 


673. 


Between 61 and 80% 


34 


1,858^ 


Greater than 80% 


48 


2,611 


« ^ Total 


100% 


5,407 



On the great bulk of social studies exercises the panel was 
"realistically satisfied" withi^ proper ^responses from no 
less than 60% of the population. In almost half of the exer- 
cises the panel wanted proper responses from 80% of the 
population. # 

Summary 

The data presented here are summary dajta;.they hide 
and gloss over many fine points. For example, on some 
single exercise the panel may h^Ve judged the exercise of 
high value in terms . of desirability but was realistic enough 
to be satisfied with a fairly low percehtof proper response^. 
Best interpretations^areji^ from <xarr.:r/lng the panel's 
Tatinij of each exercise. Such examination fits with NAEP's 
point of view that each exercise should be examined by 
itself. ^ . 

With suitable limitations in mind, it appears from panel 
ratings that National Assessment exercises are closely 
related to the NCSS Social Studies Curriculum Guidelines (with 
the exception of the component, social participation) and 
possess some obvious degree of desirability according to 
^anel judgments from the*guidelines. Professionals would be 
realistically satisfied if 61% or more of the population gave 
correct responses on most social studies exercises. 



mTES 

1. Suitnl Sludir^ Citmculum Guidelines (Washington. D.C.:, National Council for the Social Studies, 1971). 

2. Appendix D. Reference 14. 

3 See :\ppendix \X Reference 5." [v^ 
.4. See .\ppcndix O, Reference 8, p. 14 (Exercise RSi 18). Released exercises will be published in full in succeeding 
. /^ NAEP reports. 

\\ ^ See Appendix D. Reference 8, p. 52 (Exercise RAB02). 
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CHAPTER 8 

AN ANALYSIS OF A SELECTED SET OF SOCIAL STUDIES 
EXERCISESrKNOWl,EDGE OF INSTITUTIONS 

C. Benjamin Cox 
University of Illinois 



Introduction 

One of the major tasks identified for the overall assess- 
ment of tne national assessment of social studies by a^ 
researcher group from the National Council for the Social 
Studies (NCSS) was an interpretation of the data generated 
in the citizenship assessment of 1969^0 and the social 
studies assessment of 1971-72. The expecfetion was that ways 
could be found to compare the findings from the two assess- 
ments since they shared areas of interest. Perhaps a confir- 
mation of citizenship findings or even some indication of 
growth between the two assessments would somehow fall 
out of the comparisons. 

It was further expected that a number of different group- 
ings of the data would allow some useful embellishments in 
our interpretations. In general, these groupings require the 
selection of items from the total assessment to be treated 
together. The going idioms at the National Assessment of 
Educatioifal Progress (NAEP) for such a group of items are 
**color scheme*' and ''theme?' The computer is asked to put 
the items together in oMer to compute all manner of 
statistical results, which -presumably say more to- the 
researcher about the population he is working with than if 
he had results only from individual items. The options in the 
NAEP model are individual item data and data from * 
clusters of items or '*color schemes.** ' 

The Intention in the NCSS assessment was to use a 
propitious combination of these two options. A variety of 
color schemes was envisioned ~ e.g., factual items, value 
items, skill items, historical items, sociology items, 
economics items, etc.; items grouped according to the NCSS 
guidelines; items grouped by NAEP objectives in social 
studies; items grouped by a variety of content themes other 
than the mother disciplines, e.g., raciaj concerns, moral 
questions, civil rights; etc.; arid items assessing critical 
thinking, logic, judgment ^nd decision making, reasoning, 
etc. 

There was also the intention of casting explanatory 
hypotheses in an effort to account for good and bad perfor- 
mances. There was no expectation that such hypotheses 
could be tested, of course, since National Assessmem data 
are purely descriptive. This intention would be greatly 
enhanced by the data breakdown built into the NAEP treat- 



, ment. The treatment involves the breakdown of data by ages 
9,.13, 17 and adult; Northeast, Southeast, Central and West 
regions of the U.S.; two sexes; two races; four levels of 
parental education; and seven size and type of community 
(STOC) categories. Thus, the seeker of explanations is 
guided to ask, ■* Why do blacks perform worse than whites 
on this item about the Supreme Court?** When a cluster of 
items in a color scheme is under consideration, he may ask, 
"Why do persons in the upper socioeconomic areas of the 
cities know more about their democratic rights and duties 
than persons who live in rural areas?" 

A 'further intention within this interpretive task was to 
draw implications for public policy. For example, one is 
tempted to infer from the consistently poor performances of 
blacks in this assessment that something is awry in the con- 
duct of schools, fn curricul|jas, in teaching strategies, in the 
society, in the assessment procedures, in 'the assessment 
content or possibly in the black population itself. Some of 
these, at least, would be subject to alteration by changes in 
public policy. 

A final intention was. to translate findings and data into 
forms more useful to an assortment of users. Wl^ile the pro- 
ject's treatment of data is not at the sonhistiratcd statistical 
level of much of education research, it is several cuts above 
simple addition and subtraction. Multivariant analysis, 
regression formulas, correlation coefficients and chi squares 
are absent here; but there are sufficient NAEP idioms, such 
a raw p, delta p, cut-off,^hinge and eighth as well as stan- 
dard terms in the nomenclature, such as category, variable, 
standard error, median and mean, to boggle the mind The 
craft of the interpreter, one would^ suppose, is to say obtuse 
things in ways that are meaningful to readers. 

Some Limitations — Fortuitous and Otherwise 

\ We have suggested some of the limitations of this in- 
terpretation. Chief among these is the impossibility of 
finding correlations between-a performance in one^ituati9n 
and a performance. elsewhere- Educators are fond of dis- 
covering such relationships — e.g., between I.Q. and 
reading or belief in x and belief in y. The reason for .this 
limitation is the sampling and testing schemes employed by 
National Assessment. As more elaborately explained in 
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Larkins* chapter, NAEP'has constructed sample^ popula- 
tions of some 27,000 persons in each age Tevel of 9, 13 and 
t7, and 9,000 adults. However, out of defciTnc« for the 
physical and mental comfort of these persons, the total 
number of items being used to assess a learning area is 
divided into 10 to 14 sets or packages so that each package 
' contains only 1/10 or 1/14 of«the items. An individual or 
group selected for testing as a part of an age sample would 

-receive only one package containing, say, 20 items out of a 
total of 200 to 300 ip a learning area, llius, the items con- 
tained in a given package^would be answered by only 2,000 

. to 2i500 persons or so. Correlations between items within 
packages would be possible since these items would have 
been answered byTRe same persons. While NAEP has ex- 
perimenter! with this further treatment of assessment data, 
the data ^ resented here arc not dealt with in this way. Cor- 
relations between items in different packages are not 
* statistically permissible, however, because the two sets of 
items would have been answered by two different groups Of 
people* 

Statisticians substantiate that when appropriate sampling 
methods are used, clusters of items^ as in a color scheme, , 
may be treated aS; if they represent the performance of a 
total age level population. Thus, answers to a cluster of 10 
items^ while possibly the actual'performances o? 10 groups 
of 2,500 pefsons,' each group having ^nswered one item in 
^he cluster, may legitimately represent the performance of 

^ the .entire sample. An assumption of group equivalency i& 

^ made in this regard^ of course. 

A second serious limitation relates to a series of fortuitous 
events that range temporally over a year and a half, 

' geographically from Washington, D.C., to San Francisco, 
and ethically from principal to expediency., The upshot of 
the di^culties is that severed budgets, delayed computers 
and contract deadlines have restricted this interpretation to 

\. fewer than 50 items out of some 200 in the social studies as- 
^sess^jmeht and to two color schemes out of a planned half* 
dozen or so: We anticipate that in the future this interpreta- 
tion will be completed with all social studies items included 
and , with whatever adclitional color schemes are on- the 
NAEP computer tape. 

We hope to correct another limitation in that future ex- 
pansion of this interpretation. This document contains only 
limited references to the earlier fmdings in the citizenship 
assessmem. An unfortunate deterrent to any comparison of 
. the results of the two assessments is the fixing of summary 



data i.e., the color schemes or clusters — by 
rubrics. A presumnption is made, however, that some goClsin 
the two areas are so similar that compacisons are ^Tossible 
and that spme items in the two assessments ca^/oe found 
that appear to ask similar things. 

Items included. This interpretation is Uotited to the 
consideration of the items dispigyed in Appendix C, where ^ 
they are grouped into two color scheraes^ Tlie two schemes, 
originally requested for this NCSS assessment in the 
categories of "Knowledge of Rights* and Duties'* and 
'"Knowledge of Institutional Structures," largely parallel the 
social studies items used in NAEP*s recent publication 
Political Knowledge and Attitudes 7977-72, 

Reporting the Data . i 

All data, all^.statistical treatments of data and all in- 
terpretations of statistics contain inherent distortions and 
inaccuracies. The fact is that statistics, statisticians and in^^ 
terpreters of statistics impose their own peculiar frames on 
data. The danger occurs in the pretense of purity, which is 
not and cannot be. That pretense is not made about the dat^a 
and interpretations reported here. 

For the most part, percentage of success is the basic statistic 
used in the Assessment.. That means simply the percentage 
of persons from a desigi^^ted population who marked the 
item correctly or successfully, as judged by NAEP. This, 
statistic is called raw p in NAEF jargon; we use raw p, 
percentage and percjpntage of success as equivalent terms. 

Another National Assessment convention derived from 
two percentages of success is also used in this report. One of 
th^se values is the national percentage of success on a given 
exercise. Tliis value always refers to the performance of all 
the persons in an age level — i.e., 9, 13, 17 or adult — who 
responded to an exercise, usually about 2,500 persons, give 
or take 200 to 250. There is never a combination of ages in 
^ this or any statistic in reporting NAEP data. The other 
percentage of success used refers to the performance of a 
group within the exercise population) e.g., males^ females; 
blacks, whites; persons living within the Northeast, 
Southeast, Central or West region. The dificrence between 
the national percentage of success (raw p) and the group *s 
percentage of success (raw p) is reported as delta p, 

The values referred to thus far are illustrated in Table 5. 
The data are presented as representing the performance of 
i 3-year-olds on exercise number 406011. 





TABLE 5. Sample 


Exercise Data - 


- 13-year-old Respondents 








National 




Regio 


n 




Sex 








Southeast 


West 


Central 


Northeast 


Male 


Female 


Raw p 


73.7% 


69:9% 


74.2% 


74.1% 


76.3% 


74.4% 


73.0% 


Delta p ^ 




-3.8 0 


0.5 


0.4 


2.6 


0.7 


-0.7 


N-count 


2,687 


706 


660 ' 


674 


647 


1,381 


1,306 



Thus, 73.7% ofAhe 2,687 13 -year-old respondents who 
marked this exercise, answered it correctly. Of the 706 13- 
year-old respondents who lived in the Southeast region, 
69.9% answered the exercise correctly. The Southeastern 
13-year-olds* percentage of success is 3.8 percentage points 
below the national percer^tage of success, and so on. Delta p- 



vajues will be used in several displays in this report when 
referring to individual exercises.. 

Another statistical arrangement is used when dealing 
with summary data, i.e., groups of exercises *J>r c^lor 
schemes. Fn this case, all the exercises in the color scheme 
are ordered from highest to lowest delta p-yalues of a group 
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whhinsan age IcvcL The median value in this list of delta p's 
is taken as the typical performance of the group on, that set 
of exercises. Thus^ex^rcises in the list whose (fcjta p-values 
are at or near this median value arc said to beiypical of this 
group^S performance in the domain of the color scheme. A 
Complex statistical operation is employed to ^eterminci on 
the bas?s of such information a^ the standard error and the 
difficulty of each exercise for the group, which of the exer- 
cises ^n the color scheme were answered atypitrally by the 
group. Another set of statistical functions is performed to 
determine whether a given delta jp-value is large enough to 
be called significantly or reliably different than the national 
raw p. Thus» when dealing with color schemes, the com- 
puter progpam gives to the interpreter a list of exercises, 
ordered by the deha p-values for which a median has been 



determined, and in which performances atypical of the 
'*group effect'' have been demarcated and performances 
statistically signilicantly different thaja the national percen:. 
tages have been fiagged. These clem6nts are. boons to the in- 
terpreter since they give him something extra to s^y about 
the group's performance on the set of exercises, i.e., that is 
performed cpmp^atively better, or worse than expected 
(atypically) on the dehiarcated exercises relative to its per- 
formances on the other exercises and that it performed 
significantly better or worse than the whble age Sample on 
the flagged exercises. Our use of the typical and significant 
nomenclature will remain consistd^nt throughout the report, 
e.g., significant will always mean statistically significant. 

An example of such a display of a color scheme is shown 
in Table 6. 



TABLE 6. Color-Schctnc Display of Exercises and Performances 

Age: 9 Size and Type of Community: Rural 

Color Scheme: Knowledge of Institutions (Structures and Rights and Duties) 



Delta 
P- Values 

4.20* 
3.03 



0.33 
(median: 0.220) 
0.11 



-4.83 
--8.49* 



National 
Percentages of 
Success (Raw P) 

88.93% 

35.92 

83.18 

73.82 

57.54 
46.85 



J- 



In this short color scheme, rural 9-year-olds performed 
typkalfy at a level very near the national raw p. The perfor- 
rnances on the two exercises nearest the hypothetical me- 
dian (medians must be computed in even-numhered lists in 
Y)rder to make a 5a-50 split) are only .11 and .33 above the 
national raw p. A group *s typical performances db not have 
to be clos^ to the national raw p's, of course. The reference 
point for the judgment of typicality is the group's own me- 
dian, the "most typical" performance in the color scheme. 
Typical performances are contained within calculated limits 
on either side of the group's median. It is possible for all c,^a 
group's performances to lie within the limits of typicality; 
ordinarily, however, at least a few extreme scores in a 
group's dispersion on a color scheme are judged to be, 
atypical. The best performances in the example, in which 
the rurai 9-year-olds are 4.20 and 3.03 percentage points 
above the national raw p's, are better than their typical per- 
formances. Likewise, their worst performances, 8.49 and 
4.83 percentage points below the national raw p's, are worse 
than their typical performances. 

Furthermore* the two extreme deha p'-values of -8.49 and 
4.20 are flagGjcd by the computer as being significantly dif- 
ferent than the na'tional percentages for the exercises. The 
two delta p-values Sf -4.S3 and 3.03 are merely chance 
variances from the national percentages. While statistically 
significantly different scores normally appear at the low and 
high ends of a dispersion, significance is a function of the 
size of the delta p score on an exercise, not of the dispersion 
/;^r A/. A group's performances on all the exercises in a color 
scheme could be significantly better or worse than the 



Exercises: Short Text 
Unreleased text 

Which Job Heahh Dept. - Inspect, kistaurant 
UriVeleased text 

Who Responsible Fair Trial/Judge 

Who Head Town Government/Mayor 
Unreleased iext 



national performances on the exercises. Thus, a delta p- 
vaiue may hfi; typical and significant, typical and not signifi- 
cant, atypicarand significant, or atypical and not signifi- 
cant. • 

The wily reader of statistics will have inferred by now 
that an assumption of relatedness of exercises must be made 
to support the assertions of typicality and atypicality in all 
cases. The existence of a valid color Scheme is the founda- 
tion of all such group summaries. Statistically, the most 
valid color schemes would be produced by factor analysis. 
Then one could tell operationally which exercises go 
together. National Assessment hai,toyed with this notion, 
but color schemes to date, including the two involved here, 
have content validity only., That, too, could be enhanced 
whh the help of a few like-minded experts. Hunkin's chapter 
offers a surfeh of visual judgments on which exercises in the 
social studies and citizenship assessments are valid can- 
didates for their designated offices. Also Chapin's chapter 
reports jury decisions that relate to exercise validity. These* 
materials were not available, however, when the judgments 
on these two color schemes were majje. To the,«xtent that 
this researcher's decisions efiected mese groupings, no ap- 
peal was made to a wider expertise. That lonely culpability 
notwithstanding, the schemes are not precisely as re- 
quested; so help came frqm somewhere. 

A final comment on the presentation. of group data in a 
color scheme concerns -the dispersion of deha p-values on 
either side^of the typifying median. Earlier we explained the 
Assessment's convention of u^ihg the median value in an 
ordered array of exercise performances as the most typical 
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performance of a group. It was also suggested in that discus* 
^sion that pegging the "group effect," "median value" or 
'"typical performance'* of, say, t3*year-oId blacks at 11.22 
percentage poix^s below the national percentage of success 
for all 13-ydar-olds is a useful but incoiyplete bit of informa** 
tioh. The mediatji delta p tells us nothirig about the disper* 
sion of delta p-values above and below that median value. 
* The metfrair"valuc~ ewnnpfer dted above (delta p = 
—11.22%) will illustrate the usefulness of also noting the 
range or dispersion of delta p-values.,in the color scheme in* 
volved in that instance, there are 26 exercises* For'the entire 
listing of 26 exercises, the group effect (median delta p) of 
13-/ear-old blacks is 11.22 percentage points below the 
national r'aw p. However, the poorest performance of this 
group on any exercise in the list was 28.98 percentage points 
below the national raw p» At the other end of the scale, the 
best performance for the group was on an exercise where it 
scored 7.52 percentage points above the national raw p. The 
median and range can be shown in some such display as the 
following: 

Lowest Delta P Median Delta P Highest Delta P 

-28.98 -11.22 +7.52 

While none of the exhibits in this chapter include this dis- 
persion information, frequent and consistent references are 
made to the extreme limits of .the distributions in the in- 
terpretive discussions. The highest and lowest delta p-values 
dre often atypical in that they depart farthest from the me- 
dian delta p. 

Limitations of the Data 

What with all the complexities of treatment, one would 
presum^ that a wealth of hypotheses could be invented 
about youth, learning, schools and cultural effects in this 
country from NAEP data. Numbers of observations and 
complexities of treatment notwithstanding, * the wealth 
available is extractable primarily through the loosest kind of 



■ / . • . 

inferring. In the statistical world, trade in loose inferences is 
bearish. Moreover, National Assessment, being a political 
creature, is constrained from engaging in such creative 
hypothesizing. , 

More specific to the point here, it is the nature of the data 
that poses the harshest limitations. Primarily what we have 
here are disjunctive data, to coin a phrase^ rather than con- 
junctive data, to-eotn^ttothert and they are disjunctive in 
several ways. ^ . 

^ "First, as explained earlier in this chapter, the grand sam- 
ple of,: say, 25,000 persons in an age level turns out in the 
real \\rorld to be a composite of, say, 10 samples of 2,5(K) 
persons each. An assessment in a learning area may be com- 
prised of 250,eXerGises. However, no one person and no one 
sample takes that sort of test; instead, in the real world, each , 
works with only 25 social studies exercises. Later they are 
quilted together to compose the social studies assessment. 
Given the right assumptions about sample randomness, test 
reliability, exercise validity, etc. and the right and teroperly 
rigorous statistical controls, this disjunctive aprpr<^ch can 
produce some reliable knowledge of a^gross sort. This sam- 
ple and test disjunction needs to be kept in mind jion-' 
etheless, o 

There is also a disjunction across ages that makes 
generalizations comparing one age level with another 
tenuous. The problem is mainly one of test equivalency, 
though all the sampling problems dealt with above are ac- 
tive here too., While the big goals and objectives in social ' 
studies are shared by all the age levels, common sense alone 
dictates different translations of these goals, different ac- 
tivities, different content, etc., at least to a degree. Thus, 
while all age levels may share some questions, each level 
may have a unique combination overall and some unique 
questions of its own. A simple display of the '''Rights and 
Duties** color scheme (Table 7) utilized in this p^per will il- 
lustrate the problem. The color scheme as a whole ?s com- 
prised of 22 exercises, variously shared by the different age ^ 
levels, V ^ 



TABL£ 7. Exercise Distribution iii Rights and Duties Color Scheme 



Exercise: 


1 2 3 4 5'^ 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 16 


17 ^ 18 


19 20 , 21 


22 


AgeO 


X 




























Age 13 


X X 








X 




X 


X 


X 


X 


X 




X X 


X 


Age 17 


X X X X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X X 


X X 


X x/x 


X 


Adult. 


X X X X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X X 


X X 


X X 


X 



Cross-age conclusions between 17-year-olds and aduUs 
present the least difflcUhy; the color scheme for them is sub- 
stantially equal. But cross-age inferences between ages 13 
and 17 must be drawn from half the number of exercises, a 
dubious equation at best; and nothing useful can be said 



about 9-year-old respondents. 

Though the data ate superfluous to the point, we will dis- 
play the exercise sharing for the other color scheme, 

Knowledge of Structures'* (Table 8), as a handy 
checkpoint for interested readers. 



TABLE 8, Exercise Distribution in Knowledge of Structures Color Scheme 



Exercise: 


1 2 3 


4 5 6 


7' 


8 


9 


10 


11 


12 


13 


14 


15 


16 


17 


18 


19 20 


21 


22 


23 


24 


25 


26 


Age 9 


X X 


X X 


























X 








Age 13 


X 


X 


X 


X 


X 












X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Age 17 


X 


X 


X 


.X 


X 


X 


X 


X 


X 


X 


X * 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Aduh 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 




X 


X 


X 


X. 


X 


X 


X 


X 
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As wc have s^howh in several ways, though without a for- 
mal layout, NAEP data arc broken down, for each age level 
in a variety of interestiiig and pertinent ways. As a starting 



point for this discussion, we will display these breakdowns 
in Exhibit 2. 



EXHIBIT 2. Cfttegorical Breakdown of NAEP 
Assessment Data 




National 
Itegion 



Sex 



Southeast 
West 
Central 
Northeast 

Male 
Female 



Race 



Black 
White 
Other 



Parental Education 

No high school 
Some -high school 
Graduated high school 
Post high school o 
Unknown 



Size and Type of Community (STOC) 
Low metro 
Extreme rural 
Small place 
Medium city 
Main big city 
Urban fringe 
High metro 



These categories, with the help of the inset map of the 
United Stat^, arc prcsunmably Self-explanatory ^yith the ex- 
ception of size and type of community (STOC). Because the 
definitions of these seven categories arc fairly 

technical and involved, the National Assessment designa- 
tions follow. Readers arc lirgcd to check these definitions 
with seme care since they represent a propitious blending of 
geographic and. socioeconomic concepts. Please note that " 
the STOC categories apply only to repondents in school at 
the time of the assessment. , 

Low metro. This category "comprises 7.25% of the total- 
sample and represents individuals attending schools whose 
students have the largest proportions of parents not 
regularly employed and/or on welfare. 
Extreme rtml. This category comprises 9.15% of the total 
sample and represents individuals attending schools whose 
students have the largest proportions of pai^cnts engaged in 
farm work* 

Small place. This category comprises 28.29% of the total 
sample and represents individuals attending schools in com- 
munities with populations less than 25,000 and not clas- 
sified under extreme rural. 

Medium city. This category represents 17.40% of the total 
sample and represents individuals attending schools in com- 
munities with populations between 25,000 and 200,000 and 
not classified in low metro, extreme rural or high metro. 
Main big city. This category comprises 9.41% of the total 
sample and represents in4ividuals attending schools within 
the city limits of cities with populations greater than 200,000 
and not classified in low metro or high metro. 
'Urban fringe. This category comprises 17.35% of the total 
sample and represents individuals attending schools in 
metropolitan areas served by cities with populations greater 
than 200,000 but outside the city limits and not classified in 
low metro, extreme rural or high metro. 
High metro. This category comprises i2.15% of the to|al 
sample and represents individuals attending schools whose 
students have the largest proportions of parents in profes- 
sional or managerial positions. 

Now, back to our main point about the disjunctivencss of 
these data. Each of the age levels, 9, 13, 17, adult, is_ 
categorizable in each of the "ways sHown in the exhibit. 
Thus, we can show the percentage of success (raw p) of all 
9-year-olds (national), for example, on any exercise des- 
ignated for 9-year-olds and the group effect (median raw p) 
and dispersion of all 9-year-old performances across a color 
scheme of exercises' Further, we can show these same 
statistical performances (raw p, median raw p, etc.) for 9- 
year-olds who live in the Southeast, or who are males, or who 
are black, or whose parents have not gone to high school, or 
who live more accurately, who go to school — in a low 
metro community. The repeated use of the disjunctive or in 
the sentence dramatizes the disjunctive data problem. We 
cannot show by these data the performance of Southeastern^ 
black, male 9-year-olds whosfi parents have not gone to high 
school and who go to school in a low metro community. In 
fact no cross-category combinations of any kind are 
available in these data. 

This is not a limitation of which NAEP is unaware. It has, 
in fact, contemplated future assessments in which some such ' 
combinations are entailed. On a national scale, however, the 
difficulties are awesome; on a regional scale, they are only 
horrendous. A major part of the problem relates to sample 
size and to the inferring of population facts from sample 
data. A hypothetical case will illustrate the point. Suppose 
there are 75,000 9-year-bIds in the sample. Only 2,500 of 



these youngsters would be adniinistcred a given exercise. 
Perhaps 660 of this group would live in the Southeast and 
only one half of these, 330, would be males. In the Southeast 
perhaps one third, 1 10, would be black and certainly fewer 
than one half, 55, would have parents who never attended 
high schcoL Finally, possibly less than one fifth of these , 
would be categorizable as low rtietro. That would provide a 
sample of a dozen or so, and possibly fewer, from which to 
infer population' facts. The confidence level of one's 
statistical inferences would be low. Whether, for all the 
bother, it would improve on pure guess is open to specula- 
tion, ' 

With this examination of the data-producing machinery 
and the nature of the data produced, we.will turn to an ex- 
amination of the data themselves. Our displays and discus- 
sions will make use of the concepts and information ex- 
. plored in this first part. 

A R^ :iew of the Data 
* in the National Assessment of ' 
Social Studies, 1971-72 

We begin our discussion of the results of the social studies 
assessment with reference to the three basic displays of 
group effect m Exhibits 3, 4 and .5. The values shown in 
these three exhibits arc median raw p-values. In Exhibit 3 
the group effect is shown for each age level in the categories 
of national; Southeast, Northeast, Central and West region; 
black and white race; high and low metro STOC; and no 
and post high school parental education over the entir? set 
of exercises included in this report. These include 6 exercises 
for 9-year-'oIds, 26 exdfcises for 1 3-year-olds, 42 for 17-year- 
blds and 41 for adults. This total group of exercises is 
presumed to reflect the respondents' overall "Knowledge of 
Institutions." 

Exhibit 4 summarizes the group effect for each age level in 
the same categorie^s as in Exhibit 3 for the color scheme, 
• **Knowledge of Institutional Strucfures. Exhibit 5 displays 
median raw p-values for the color scheme, "Knowledge of 
^.^ Institutional Rights and Duties. " The results for the "Struc- 
ture" theme dre based on '5 exercises for 9-year-oIds, 16 ex- 
ercises for l3-year^oIds, 21 exercises for both l7-years-olds 
and adults. The "Rights and Duties" theme results arc 
^ based on 10 exercises for 13-year-olds, 21 for 17*.year-oIds 
and 20 for adults. No 9-year-old results are possible here 
because only one exercise was judged to treat this theme. 

Generalizing from these data in Exhibits 3, 4 and 5 is a bit 
hazardous; but to he perfecdy honest and open, one must 
say that for every display. Rather than repeating it endless- 
ly* we wil! assume that alKwill proceed from this point with 
caution. ' 

The striking feature of the three graphs is the general con- 
figuration of lines. If we cpuld make an assumption of test 
equivalency across the four age levels — which, of course, 
we cannot — we could say that the years between 9 and 13 
are something of a loss for young persons in the U.S. as far 
as this aspect of social studies is concerned; but the social 
and political world comes alive for them sometime after age 
13. By age 177 then, most will have learned all they ever will; 
in their adult life as many will lose as will gain in knowledge 
of jLh^ political and social institutions that govern their lives 
and their rights and duties within them. While a number of 
(constraints will not allow us to basie so strong a statement on 
these data alone, we suspect the hints are indeed there and 
something like this is in fact the case. Even discounting the 
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Median 
RAW P Z 

100 



AGE « ft • 

NINE 



THIRTEEN 



SEVENTEEN 



ADULT 




POST H.S 
HIGH METRO 



WHITE 

NORTHEAST 

CENTRAL 

NATIONAL 

WEST 



SOUTHEAST 

NO H S 
LOW METRO 



BLACK 



A6 - - , - - 

- - - 

42 - - 

40 ^ „ „ 

EHHIBIT 3t Knowledge of Institutions - Median Raw P-Values 
Nation, Region, Race, STOC, Parental Education - 
Social Studies 1971-72 
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Hedianv AGE.;. 

RAW P 2 NINE 

100 - 

98 - 



THIRTEEN 



SEVENTEEN 



96 - 

94 _ 

92 - 

90 - 



ADULT 




POST H S 
HI METRO 



WHITE 

CENTRAL 

NORTHEAST 

NATIONAL 
WEST 

SOUTHEAST 



NO H S 



LOW METRO 



BLACK 



EXHIBIT 4. Knowledge of Ins'titutional Structures ■ 
Median Ray P-Values - Nation, Region, Race, STOC, 
Parental Education - Social Studies 1971-72 
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Median AGE..\ 

RAM P Z NINE 
100 

.98 - 

96 - 

94 - 

92 - 

90 - 

88 - 

86 - 

; 84 ._ 

82 _ 

80 _ 

78 - 

76 - 

74 - 

72 - 

70 - ' 

68 - 

66 - 

64 - 

62 - 

60 _ 

58 _ 

56 _ 

54 - 

52 _ 

50 _ 

48 _ 

46 - 

I 

40 



THIRTEEN 



SEVENTEEN 



ADULT 




HI METRO 
POST H S 



NORTHEAST 
WHITE 
CENTRAL 
WEST 

NATIONAL 



SOUTHEAST 
NO H S 

LOW METRO 
BUCK 



EXHIBIT 5. Knowledge of .Institutional Rights and Duties 
Median Raw P-Values - Nation, Region, Race, STOC, 
Parental Education - Social Studies 1971-72 
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results ai age 9, which are probably spurious due to the few 
exercises, we are left with the dramatic rise from age 13 to 
age 17 and the noticeable cooling offafter age V. The sets of 
exercises given to 13-year.olds and H-year-olds are only 
marginally comparable, but there is near equivalency 
between the 17-year-old and aduh exercises. A similac^Jiut- 
less pronounced, rising and cooling effect is evident in the 
earlier citizenship assessment for some groups^ principally 
males, medium size cities, urban fringes and amaller'places, 
on a similar theme. ^ ^ 

So, by rounding, squinting and allowing in several ways, 
we can probably say that something is fairly successful in in- 
creasing young persons' knowledge of institutions by as 
much as 25%, in relative terms, between ages 13 and 17 (an 
absolute gain of about 15 percentage points in these data). 
We may also be able to say that schools in general and social 
studies iri particular play some part in this growth.^ Most 
students have almost two years of U.S. history and one year 
of government between ages. 13 and 17. That docs not sug. 
gcst that there are grounds for pride in this rough calcula- 
tion, however, A gpod part of the increase may be accounted 
for by Piagetian hypptheses concerning the general shift 
from the concrete operations stage to th^ forrrial -operations 
stage of cognitive development in this age range. Under^ 
standing of this content and interest in it may be as much 
products of development %s schooling. 

We infer the effect of, schooling on what apparently hap- 
pens to persons when they leave school. They retain, at least 
in part, what they learned as teen-agers 10 years or so 
earlier; but they do not continue to grow. Now, of course, a 
statement like th^t is not warranted £y these data. We know 
absolutely nothing about these particular young adults 
when they were 17 except by inference and conjecture. 

They were born between, say, 1937 and 1947 and half or 
more would have graduated from high school between 1954 
and 1964 at about 17 years of age. Schools were not much 
different than from 1971-72 when this assessment was made. 
Furthermore, these young adults knew something of World 
War II, or at least its aftermath, and lived thrdugh Korea and 
Viet Nam; :some served in the armed forces there or 
elsewhere. They saw their president and his assassin killed 
on television and watched the whole panorama of strife and 
dissent throughout the sanguine 60s. Some would have, 
voted a dozen times perhaps. 

Therein lies a tragedy that these data suggest. Learning 
in this country^ at least as it measured by these exercises and 
as it concerns this content,, stops at the schoolhouse door, 
not, as some have insisted, on the way in, but on the way 
out. Schoolmen still have no certain grounds for pride here, 
however; for what is learned in schools and, consequently, 
what is being assessed here by NAEP may be so irrelevant to 
a citizen's life purposes and activities that he has no reason 
to learn any more of it. Also what is*'^being noted here as ' 
learning may be a consequence of experienc<^s at a certain 
z^ge, but not \ necessarily school experiences. 

Another tragedy is depicted by Exhibits 3, 4 and 5. The 
gross difTerences and inequalities in our society that attend 
race and class, for ^the most part, are part and* parcel' of 
schooling, learning and testing. That, of course, is hardly a 
startling revelation, "Everybody knows thai the racist and 
classist biases in this society are pervasive. They operate in 
schools quite as easily as anywhere, despite 20 years of 
desegregation experience. Howevjr, buried in the tangle of 



lines in Exhibit 4 is the whisper of a hypothesis that, assum- 
ing a part of the difference between t3^d 17 is a schooling 
effect, blacks and the poor may geUwJmething of a break in 
schools. While the blacks, the lov/metros (these are mainly 
the inner city poor and probably are largely black) and 

^ thos e wh o se p arcnts-never attended high school have the 
three lowest median raw p-values of all 13-year-oid groups 
(around 60%) in tl^5 Rights and Duties color scheme, they , 
sliow the most precipitous gains by age 17. The low metro ^ 
groups show a 20 percentage point gain between age 13 and 
age 17; the black samples and the no high school samples 
show^ 13'- and 15-point differences. The national difference is ' 
only 4 points and no other group gained more than 8.3%. 
The gains of the low metro, no high school and black groups 
may be more of a function of starting points. It is much 
easier to post gains with a starting point of 59% than it ^* 
with a starting point of 77%. Their gains are short livcd^-^^ 
however; in the aduh vi^orld, the black and low metrb 
categories, along with the Southeast region, show group dis- 
advantages of from 6 to 12 percentage points below the 
national median. . ^ ^ 

An alternative hypothesis may be more powerful. These 
l7.year-old blacks in 1971-72 had been witness to and had 
probably participated in a dramatic civil rights revolution in 
our society throughout their school life. They, more than 
most, wpuld have been sensitized to such issues since 1954, 
the year in which some of them were born. In this 
hypothesis,* the schools can claim no credit, for it was these 
young 17;*year-olds who brbugiu^the pertinent concepts to 
the school; they didn't learn themlhere^ 

There is also the possibility that the gains for black, no 
high school and fow metro groups reflect a condition noted 
in other studies — e.g., the Columbia University Citizenship 
Education Project — > that black youths and Some others 
learti more about politics and the like in school because they . 
have less opportunity for formal learning about institutions 
outside of school than do white, middle-class youths and 
some others. ■ ' 

», - ^ - ^ • 

Comparfson of Group Differences from 

National Success Levels in Social 
Studies and Citizensliip Assessments 

In the following exhibits and discussions, group perfor- 
mances will be reported as delta p-values, for individual ex- 
ercises and median delta p-values for groups of exercises or 
color schemes. When delta p-values are used, the perfor- 
mance of a selected group is shov/n as deviating by so many 
percentage points above or below the national raw p-value 
on a given exercise. In the exhibits in this section, we will 
adopt a NAEP convention by showing the' national perfor- 
mance levels as iying along a horizontal line designated as 0. 
It is not useful to identify national raw p-values in the ex- 
hibits since the median delta p>values charted in the figures 
for fwo or more groups at any age level most likely report 
scores on different exercises. Exhibits simply display the 
typical performances of groups. The performance level of 
the groups involved will be charted as broken lines that con- 
nect the median delta p-values for e^ch age level above and 
below the 0 national line. Exhibit 6 is an example of this 
convention. 



58 

00066 



E^ailBiT 6. . Knowledge of Institutions - 
Regions ^ Median Delta P ^^ 




3 



ABOVE 1 
0 

% 

-2 

BELOW 

-3 



In Exhibit 6» the Central region 9-year-oids are 2 percen- 
tage points above the national raw p on their median exer* ' 
cise while Western ?-year-oId^ are 2.1 percentage points 
^ 'betow the national level on their median exercise, etc. In our 

, subsej^uent discussions^ we may, for convenience and relief, 
i«se shortened versions of such statements, e.g., Central 9- 
ye^r-olds aire 2% above the national percentage while 
Western 9-year-olds are -2.1% below. ^ 

Our first set of exhibits will depict |he performances of * 
selected groups on the total number of exercises included in . 
this chapter. The reference* will be to "Knowledge of In- 
stitutions/' which is treated here as a cotpir scheme of 
related exercises. In four instances it hc<s been possible to 
make some general comparisons with the NAEP^s 
citizenship assessment made in 1969-70. Goal D of the 
citizenship assessment objectives^ '^Knowledge of 
Governmental Structures and Functions,'* is similar to the, 
overall color scheme, "Knowledge*of Institutions," used as 
the basis of this chapter. There are several difTicul^es with 
the comparison, not the least of which is that the citizenship 
assessment result* are reported in such enigmatic ways that 
many values simply cannot be found and all arc rounded es- 

. timates for group results. The major difficulty, of course, is 
the doubtful fit of the two sets of exercises. , 

Regions 

* * 

Exhibit 7 depicts the performances of the fom age levels 
on all exercised in the regional categories. The Exhibit 7 dis- 
play shows very little variance among the four regions, 
though on most of the exercises, the Southeastern adults are 
significantly below the national level. This is the most 
dramatic difi'erence in the exhibit, though the difference 
between Northeastern and Southeastern 13-year-olds is ex- 
actly the same as the difi"erence between the adults in the 
two regions. ^ 

Thi& real puzzle in Exhibit 7 lies with the Western 17- , 
year-olds and adults. At 17, all but 5 of the 42 exercises in- 



cluded here are b<sl6w the national percentages; one foUrth 
are significantly below. Aitiong the adults, 80% of the ej»er- 
cises are above the national percentages, and one fourth are 
significantly above, t'erhaps the high-scOring Northeastern 
teenagers ofa decade ago moved West. That hypothesis is of 
dubious quality^ obviously; at least, the pattern do<fs not 
show up in the citizenship assessment of 1969-70, which is 
displayed for regions in Exhibit 8. The same general, fairly 
flat configuration of Exhibit 7 is apparent also in Exhibit 8 
^with the Northeast and Central lines lying wholly above the 
national percentages and with the Southeast lying wholly 
belcw. The dramatic rise in the West from -0.5 at age 13 to 
17 at age 17,j5«not altogether diflerent than what occurs 
between age 17 and adult in Exhibit 7. 

Pace 

Exhibit 9 shows the performances of the four age samples 
on all exercises as categorized by black and white races. Ex- 
cluded from these displays are about 125 to 220 persons in 
each of the samples of some 2,500 persons who were clas* 
sified as "other-" The "other" category includes Uricntals, 
who qualify as racially distinct, and some ethnic groups who 
do not. The category is too indistinct to characterize ac- 
curately and too small to include in this breakdown. In 
National Assessment's testing procedures, racial 
membership is determined visually by the test ad- 
ministrators in the field. That is part of the reason for the 
confused "other" category. All field administrators do not 
share the same definitions of race. Also included here in Ex- 
hibit 10 ?>'e the group results for blacks for Goal D in the 
1969-70 citizenship assessment. Neither national percen- 
tages nor white median delta p-values are reported for the 
citizenship assessment. Nonetheless, the results are ob- 
viously tragically similar. 

As shown in Exhibit 9, the success difference between the 
' whites^ and blacks' median delta p-values at every age level 
is from 13% to over 20%. The exercise story is the same at - 
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Hedian 
Delta P 

10 

9 



AGE... 

NINE 



THIRTEEN 



SEVENTEEN 



ADULT 




NORTHEAST 

WEST 

CENTRAL 



- NATIONAL 



' SOUTHEAST 



Median % 65.6 



65.4 



80.1 



81,1 



EXHIBIT 7. Knowledge of Institutions - Regions 
Social Studies 1971-72 
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MedUn 
Delta P 
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THIRTEEN 
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BEIX)M , 

-2 
, -3 
-4 ' 
-5 
-6 « 
■ -7 - 
-8 - 
-9 - 

-10 - 

National 

Median % Not reported 




WEST> • 
NATIONAL 



SOUTHEAST 



EXHIBIT 8. Knowledge of Governmental Structures 
and Functions - Regions - Citizenship 1969-70 
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National , 

Median % 65.6 65-4 80.1 81,1 



E2[HIBIT 9. Knowledge of Institutions - Race - 
Social Studies 1971-72 
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every age level. At^ge 9, the v^hites have 6 out of 6 exercises 
significantly above the national percentages; the blacks have 
6 out of 6 significantly below. At age 13, the whites have 24 
out of 26 exercises above the national percentages, 18 of 
them significarftly above. The blacks, on the other hand, 
have 23 out of 26 exercises below the national percentages, 
with 20 significantly below. White 17-year-olds have 39 out 

^f 42 exercises above the national perct.itages whije'lhe 
black 17-year-olds have 40' out of 42 exercises below.' Of 
these, 37 are significantly above for whites and 36 ate 
significantly below for blacks* Adult whites have 39 out '6f 41 
above artd adult blacks have 39 out of 41 below the national 
percentages. AJl but <5nt delta p above and two below are 
statistically significant. On several individual exercises at 
the adult level and on a few at other levels, the difference 
between black and white raw p-values is more than 30%. In ^ 

• one case the difference is nearly 40%; a white adult, in that 
case, was two and one half times more likely to answer cor- 
rectly than a black adult. As measured "by this social studies 
assessment, the pattern emerges of two racial groups in this 
country dramatically and tragically diverse in their 
knowledge and understanding- pf the basic legal and 
political institutions that order and protect their lives and 
int^factioris. ' • » 

F^r0Q^al Education 

Exhibits 11 and 12 display breakdowns of- the social 
studies assessment results on Institutions and the earlier 
citizenship assessment results on Governmental Structures 
and Functions (Goal^D) by parental education. A respon- 
dent was placed in the post high school category if one or 
both of his parents attended some school beyond hi^h 
school; in the high school graduate category if one or both of 
his parents graduated from high school; in the some high 
school category if one or both of his parents attended high 
school, but neither graduated; and in the no high school ' 
category if neither parent went to school beyohtf the eighth 
grade. 

It is well known that father^s education has important 
social class membership effects in this society, so that kind of 
differentiation is obviously working in these arrays. A dis- 
tinctit^e characteristic shared by the two data sourcesJs the 
absolute and hierarchical discreteness of the categories 
across the age- levels. The lower-education groups never 
close the gap. The some high school adults show a 
precipitous gain in Exhibit 12, however, which would be in- 
teresting to explore. It could reflect nothing more than a 
chance sampling fluctuation; but in the midst of the paral- 
Ite^ism otherwise displayed ifi this exhibit, the results of this 
ca,tegory" across the age levels are anomolous. The parallel 
ubswings .of all adult categories in Exhibit 11 will be looked 
an more specifi^rally in the ana^ses in the next section of this 
chapter. 

^/Xe and 'Type of Commuhity 

Ejchibits 13 and 14 are also related in that they show the 
two sets of data categorized by size and type of community. 
These categories also have heavy socioeconomic effects in 
that high metro is mainly an affluent grouping; urban fringe 
is suburbanite, presumably middle-class America; extreme 
rural is primarily a farm jjopulation; and low metro is inner 
city, thus mainly, though not exclusively, city Black. 

It is tempting to overwork these data and to extend un» 
reasonable assuniptions of reliability and validity. Eut 



without invoking validity, it i^ems warranted to say that the 
exercises used in the 1969-79 citizenship assessment and 
those used in the 1971-72 social studies assessment dis- 
criminate among these STOC groups in the eight age sam- 
ples involved in very, similar ways. The four community 
^ types selected here actually include only about 45% of the 
data; but for the social studies assessment, the remaining 
results for main big city, medium city and srinall place are all 
within 1.5% of the national percentages; most are less than 
0.5%. In the citizenship assessment, all but two of the results 
not shown on Exhibit 13 are less than 2.5%. The high 
variances are included in these displays. They are un- \ 
questionably sensitive to socioeconomic influences. 

Though in the next sections, where the two social studies 
color schemes will be treated separately, we wil);^ treat the 
validity questibn more explicitly, it is transparent here, if we 
assume test validity, that the social classes in this country 
vary greatly in their knowledge and understanding of the in- 
stitutions they live by. Before this classist hypothesis is 
embraced, however, one should attend carefully to the judg- 
ments reported by both Larkins and Hunkins concerning 
the validity of objectives and exercises comprising the NAEP * 
social studies and citizenship assessments. Larkins, in par- 
ticular, suggests the presence of social-class effects in these 
assessments, but they are at least as evident in the assessors 
as in the assessed. , Schooling, we might . infer from the 
squeezing of the Variance that occurs at age 13 and age 17, 
has some effect toward leveling; but once the influence of the 
school is gone, the influences of the wider milieu surface. 
Most of the high metro adults will have gone to college; 
most of the low metro adults will not. That in itself could ac- 
count for some difference. Either the exercises are so ^ 
profoundly culturally biased that the lower classes have a 
built-in content and language disadvantage while the upper 
classes have built-in advantages; or the reading, conversa- 
tion and viewing habits, along with other traits, of the high 
and low metro groups are so diverse that matters attended 
to in one are more in line with what these exercises refer to 
while the matters attended to in the other are outside this 
realm. 

But these two explanations are the same. They both say 
that apparently the content of these questions is reinforced 
by the experiences of one group after school and not rein- 
forced by the experiences of the other. FronPi this point on 
the consideration becomes normative. One rhay say with 
Robert Coles that the effects of the ii&i<f^ city are 
pathological and possibly with James Coleman that, one set 
of values is more appropriate for life in this country; or one 
may say that neither National Assessment nor any other 
white middle-class organization — schools, for example — 
can observe, teach or assess the lower class in this country 
with accuracy and fairness. 

In the following section we will analyze the social studies 
data by the two smaller color schemes with more reference 
to individeial exercises. 

Analysis of the Social Studies Data by 
Knowledge of Structures and. 
by Knowledge of Rights and Duties 

Results reflecting the performances of the four age level 
samples and analyzed by region, sex, race, parental educa- 
tion and si^e and type of community for the two color 
schemes, Knowledge of . Structures and Knowledge of Rights 
and Duties, are displayed in Exhibits 15 through 24. The 
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Median AGE,,. ^ ^ 

Delta P NINE . THIRTEEN S]BVENTEEN ADULf 

10 - - ^ • . . 




National 

Median % 65.6 65.4 80. 1 



EXHIBIT 11. Knowledge of Institutions - Parental Education - 

Social Studies 1971-72 
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EXHIBIT 12. Knowledge of Governmental Structures 
and Functions - Parental Education - Citizenship, 1969-70 



ERIC 



66 



00074 



Median AGE... . 

Delta P NINE THIRTEEN SEVENTEEN ADULT 

. ■ . 10 - ■ • . - • . - ■- ■ . 




-13^ - 



. -14 _ - . _ 

, -15 - . . - - ■ ' 

National ' 

Median % 65.6'^' ^5.4 80*1 81.1 



EXHIBIT 13* Knowledge of Institutions - STOC 
social Studies 1971-72 
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METRO 
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EXHIBIT 14. Knowledge of Governmental Structures 
and Functions - STOC - Citizenship 1969-70 
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same statistical treatment is used, j.e., national percentages 
are shown in exhibits as 0 and group effects on color 
schemes arc shown as median deha p-values; however, each 
color scheme is based on only a portion of the total number 
.of exercises. The presumption is that each of these color 
schemesL is more precisely homogeneous than the total group 
of exercises. That, of course, is a visual judgment and r«ay 
l^e checked in Appendix C where the texts of the released ex- 
ercises and the topics of the unreleased exercises are shown. 
The results in this section are based on the following 
number of exercises; * • . - 



Stnicluret 
Rights and Duties 

Regions 



Age 9 

5 



Age 13 

16 
10 



Age 17 

21 
21 



Adult 

21 
20 



The first set of exhibits in this section is made up of Ex- 
hibit 15, which displays the Structure color scheme, and Ex- 
hibit 16, which displays the Rights and Duties color scheme 
for , each of the regional categorijrs of Southeast, West, 
Central and Northeast. 

Struetura. As E}j:hibit 15 clearly shows, the variance across 
regions for tbt exercises assessing structural knowledge is 
small. The ^^ttfja^n delta p-values are all less than 5 percen- 
tage points. Most are' less than 2%. 

At the 9-year-old level, the exercise showing the biggest 
difference between Northeast and Southeast youths asked, 
Which one of the following is usually the head of a 
government in a town? 
The mayor 
The governor * 
The chief of police 
The school principal 
1 don't krtow. 

Nationally, 57.5% of 9-year-olds krtow the correct answer; 
48. 1% of the 9-year-olds know it in the Southeast, and 66.7% 
of the Northeast 9-year-olds know it. Both values are 
^ significantly different than the national percentage and both 
are atypical responses for the regional grdUps. A simple ■ 
cultural effect may be operating in the question due to the 
different geographic profiles of the two regions, the different 
namfirs given to town heads e.g., first selectman and town 
board chairman — and even different names for towns in 
some states — e.g., villages. ^ 

At age 13, the West and Central are more alike. As pop- 
ulations they are indistinguishable, in fact^ with only two 
atypical responses and three significantly different than the 
national . percentage between them. The Northeast and 
Southeast are more different from eaich other, though the 
contrast is far from dramatic. On most exercises the two 
groups of 13-year-olds are between |5 and 10 percentages 
points apart with the Southeast in $11 but one case being 
below the national percentage and the Northeast in all but 
two cases being above. They are within 3 percentage points 
of each other on a question about financing the government. 
The text for this question is unreleased. (To provide a basis 
for future compa^rison, NAEP holds in secret about half the 
exercises used in an assessment. We make reference to these 
unreleased exercises by topic only.) At ar./ rate, fewer than 
half of all 13-year-olds know the apsWer (national pcrceri- 
tage — 47.2). Northeastern youths have a delta p of -1.6%; 
Southeastern 13-year-olds, -4.58%. The two groups are 
farthest from each other on an unreleased exercise on school ^ 



governance, Northeast 13-year-oIds are 6.2% above the 
hatlonal percentage While Southeast 1 3-year-olds. are -6.3% 
below. * 

At age 17, the two most contrasting groups are again the 
Southeast and Northeast, though the Western youths scored 
tl^e lowest of all on two unreleased constitutional questions 
to provide the greatest contrast for the Northeast with dif- 
ferences of 16 and .19 percentage points. In another un- 
released question on foreign affairs the Southeast performed 
the bestj at 5% above the national percentage, while the 
West again is lowest, at -6.1% below the national percent- 
age. 

At the aduft level, the Northeast, Central and West are in- 
distinguishable with respect to median delta. p-values. The' 
Southea$t, whose adults make the poorest showing here, is 
below the national percentages on all 21 exercises by only 
-2% to -8%. The other regions have only foUr or five results 
below the national percentages, (jone, however, is more 
than 6.8% above. ■ ' ' ' 

The range ''of performance for all adults in this color 
scheme is from a poor national p(*rcentage of success of 
45.7% oh an unreleased exercise on a constitutional provi- 
sion to a whopping 96.3% on another unreleased question 
on the armed forces. About twj^thirds of the entire adult 
sample was able to answer ^ fralf dozen unreleased ques- 
tions ori federal government details. A released question, 
typical of these deftail questions on the federal government, 
asks, ; . . , 

Which one of the following has the pow^r to declare 
an act of Congress unconstitutional? 
• The Congress » 

The President * 

The United States Supreme Court 

[The United States Department of Justice 

il don't know. 

Nationally, 61.8% of the adults answered coi'rectly; Delta 
p-value^ range from 3.0% for Centra] adults to -4.5% for 
Southe^stjcrn adults. 

Rights Ifind Duties. Exhibft 16 displays tl\e median delta p- 
values |for the four regional categories on the color scheme, 
Knowl<:dge of Institutional Rights and Duties, for age levels 
13, 17 imd adult. No 9-year^old results are referred to in this 
color sj:heme because only one exercise was classified under 
Rights and Duties. 

Rel<^ased exercises in this color scheme involve such mat- 
ters a$ atheists holding office, criticism of officials and the 
military by newspapers and others, picketing rock concerts 
and police, assembly in parks, congressmen seeking views of 
constituents, racial discrimination in employment and laws 
against vandalism. Unreleased exercises relate to freedom of 
press and religion, property and petition rights, and govern- 
ment criticism. The color scheme as a whole is presumed to 
inquii'e into knowledge and understanding of individual 
rights and duties in several institutional arrangements, 
mostly political, in this Society. 

Most of the exercises in this color scheme may be in- 
terpreted as assessing a respondent's commitment to some 
democratic values. A released example of this sort of que.g- 
tion asks. 

Should a person who doe^ not believe in God be al- 
lowed to hold a public office? 

Yes * 

No . 

Undecided. 
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EXHIBIT 15. Knowledge of Institutional Structures 
Regions - Social Studies 1971-72 
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EXHIBIT 16. ^Knowledge of Institutional Rights and Duties - 
^ Regions - Social Studies 1971-72 
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From that interpretation, it seems perfectly appropriate to 
think of the set of questions as inquiring into the democratic 
beliefs of the persons bdng questioned. Our own reluctance 
to call the color scheme by such names as attitudes, beliefs 
and commitments i$ based on two reasons. First the exer- 
cises refen for the most part, to rights and duties that are 
fully legal and established in this country. They do not pre- 
sent issues that are presently the subject of hot debate in the 
society, nor do they refer to any controversial extensions of 
democracy mto new areas. Second, it is very difllcuit to 
ascertain the actual realm of assessment with such ques- 
tions. The recent literature on value analysis only serves to 
emphasize the enigmas involved. Without, further question- 
ing and exploration, there is ho certainty whcthe) the ques- 
tion elicits descriptive or valuative information. 

To the credit of NAEP, additional information has been 
requested* on sojne of these should questions to help insure 
their operation in the value realm. The respondents are 
asked to express reasons for thsir answers. The following is a 
released e^^mple. 



TABLE 9. Northcast/Southcast/National 
Comparisons oo Selected Exercises 



A; Dojyou think people sljould be allowed to pickist 
the holding (tf a rock festival as a protest against it? 
' Yes (goloB) , ; 
No (go to B) 

Undecided (go to B) • 
No response (after 10 seconds, go to C) 
B. Please give a reason for your answer. 

Again* all we can say with respect to this strategy is that it 
is perhaps as . good as can be done with writjen exercises.. 
The furthcf" assumption is required, howler, that the 
respondent s own criterion is being expressed when he offers 
a reason. Otherwise, we are back to square one and still un- 
certain whether descriptive or normative information is be- 
ing expressedT By design, we excluded the reason giving 
parts of such questions from this knowledge color scheme in 
anticipation of having another color scheme in which they 
would be featured. This value color scheme, as we stated 
earlier* was among those not computerized. Its absence, we 
recognize, places this Knowledge of Institutional Rights and 
Duties color scheme, as well as this interpreter, in a 
vulnerable condition. 

As Exhibit 16 clearly shows, the regions consistently 
maintain their respective positions in the area of Rights and 
Duties. That suggests that the two color schemes, Structures 
and Rights and Duties, are not discrete, i.e., they are 
probably measuring similar things^ Nonetheless, the 
regional groups are more spread out in this color scheme. 
Over 10 percentage points separate Northeast and 
Southeast median delta p-values at both age 13 |ind adult 
levels. The Centr'al and Western categories at earn age level 
continue to look very much alike and continue to be the 
norming group. Only three results anjong all delta p-valUes 
for both groups for all ages are more than 5% above or below 
the national percentages. 

A selected set of exercises (Table 9) will illustrate how the 
Northeast and Southeast differ. The released questions arfr 
represented here by shortened titxts, Since the values 
reported in these examples are delta p-values and national 
percentages for individual exercises, rather than median 
values as in the previous displays, they are directly and 
specifically comparable. The Northeast and Southeast 
scores can be added to and subtracted from the national 
percentages. • 



Age 13 Agci7 
Should an atkeist hold public oifice? 



Northeast 
^Southeast 
National % . 

Shc^ld an 
publicly? 

Northeast 
Southeast 
National % * 



atheist be 



11.6* 
-16.8 ' 
59.4 

allowed 

7.6 
^-8.4 
63.9 ^ 



14.2 
-16.1 
62.6 

to express 



2.3 
-4.9 

78.1 



Should newspapers criticize public officials? 

Northeast '8.4 7^8 

Southeast -8.9 ' -9.3 

National % 48.9 73.4 



^Adult 

7.4 
-20.3 ' 
5?.6 

his views 

3.0 . 
-lO.t 
77.9 * 

2:9 
-7.0 
81.1 

Should people be allowed to picket against a rock 
festival? 

Northeast , , 6.9 ^ . 0.6 

Southeast -9.3 * -8.6 

National % 52.7 63.0 

Should people be allowed to assemble in parks to make 
demands? 

Northeast 5^6 3.4 . 5.0 

Southeast .4.6 * -7.8 

National % 63.5 87.2 77.5 



, ' The selected set of exercises above cannot be used ap-, 
propriately to characterize the groups. Exhibit 16 does that ' 
more reliably. However, some generalizations and contrasts 
inevitably emerge. , . 

In a country that is presumably built on democratic prin- 
ciples, some of those believed by many to be fairly basic fail 
in these exercises to receive anything like unanimous ap- 
proval. If one*prefers an alternative interpretation, a great 
many persons who live in this country do not know what its 
basic principles are. From another view, however, one can 
say that 75-to-80% support^for anything in a country as 
pluralistic as this one is nothing short of miraculous. 
Moreover, the application Of a value conflict model tojhese 
data would likely contribute to their understanding. 

Whatever generalization is preferred, it would appear 
fairly certain that a person^ who professes not to believe in 
Cod could not he elected dog catcher of Orange County, 
North Carolina, on a bipartisan ticket. Atheist politicians 
would not seem to be shoo-ins anywhere in the country. 
That exercise resiSIt, along with one about allowing persons 
to picliet a police station to protest police brutality and 
another about knowing that prayer is excluded from schools 
on the grounds of church and state separation, barely made 
a majority among adults. Picketing and the Supreme 
Court's religion-banning, ciajgnfared no better among the 
nation's 17-year-olds, Fewer thanTialf chose the'separation 
principle for the one and a bare majority would allow 
pickets to protest either- police brutality or a rock festival. In 
the latter cases, rejectior\ of the time honored "right" of 
protest over that range , of social offenses must mean 
.something. 
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The three age levels were asked, 
^ Should race he a factor in hiring someone for a joh? 
Yes 

■ ■ No ' . . ■ ^. . • - 
I don't know.. 

On this very important question, we take heart, Over 90% of 
every category at every age level, except Southeast and West 
r3-ycar-iolds and Southeastern adults know that racial ^is-* 
crimination in employment is iliegal or believe that race 
should not be a factor in employment, whichever interpreta- 
tion one prefers. A difference between the Southeastern and 
Western 13-year-olds is that for the Southeasterners this 
result is next to their best performance relative to the 
national percentages. They are atypically high. But for the 
Western 13-year-olds, this result is their poorest perfor- 
mance on all Rights and Duties exercises. The delta p-value 
of -6.4% is significantly below the national percentage of 
88.8% for all 13.year-olds. 

A visual comparison of the rank ordering of exercises ac- 
cording to delta p- values, a contribution of the computer 
. that will not be shared in^this report, for each of tKe age 
levels and region groups suggests, much like Exhibit 16, that 
Southe|ast and West youth arc much alike; Northeast and 
Centrajl youth have much in common; and Northeast, 
Central ^nd West adults arc very similar. The dramatic 
events that occur in this breakdown of the data across 
regions are the accommodation of eastern liberalism by 
VVest adults and the retrenchment of Southeast aduhs in a • 
conservative, Bible-belt, law-and-order posture. 

Sex \ 

Exhibits 17 and 18 display the results in the Structure and 
Rig:hts and Duties color schemes according to sex. Our first 
examination vyill concentrate on the Structjure questions. 
Structures. Exhibit 17 shows the median delta p-valucs for 
males and females across all ages foi the Structure color 
scheme. According to the median delta p-values used to 
characterize males and females at ages 9 and 13^ the two 
groups of young persons are not very different in their 
knowledge of the structures of institutions. In comparing the 
two groups at ages 9 and 13 on individual exercises, the 
similarity judgment is only partially upheld in that the two 



groups are significantly different than the national percen- 
_ tag<ii)n.7.of 16 exercises at age 13 and 2 of 5 exercises at age 
9; but in none of these is the difference more than 7 percen- 
tage points. So at ages 9 and 13^ boys and girMre different 
in what they know, believe or hold important; but they are 
not dramatically different. Differences this small maym fact 
be only chance products. 

Specifically, at age 9, males arc 5% more likely than 
females to know that mayors and not police chiefs or school 
superintendems are the heads of towns. Also boys are about 
5% more likely than girls to know that the health depart- 
ment is involved in restaurant inspection, not selling food or 
putting out fires. The importance of these pieces of informa- 
tion seems less important than the facts that only 57% of all 
9-year-olds know about town mayors and only 35% know 
about restaurant inspection. Even these results seem ex- 
plainable within the normal world of 9-year-olds. 

By age 13, males and females seem quite similar as shown 
in Exhibit 17. In fact, however, at least on some items, they 
are more different than at age 9. The males outpoint the 
females from 6 to 7 percentage points on questions relating 
to national and international governmental structures, e.g., 
what the United Nations does and how a presidential can- 
didate is nominated in the United States. But even with, 
their superior performance in the latter case, only one out of 
five of the boys knows about national political conventions. 

By age 17, a pattern that looked possible at age 13 seems 
more established. The depoliticization of females is in full 
progress by age 17. While the differences between males and 
females on individual questions are not great about half 
the significant differences are between 5 and 10 percentage 
points — they are significantly different on 14 exercises. On 
13 of these the males are above the national percentages 
while the females are below. The greatest male advantages 
arc on exercises referring to such matters as the nomination 
of presidential candidates and the declaring of congressional 
acts unconstitutional. The only significant female advantage 
is posted in a set of exercises requiring the interpretatidrfof a 
replica ballot, as follows (national percentages of success 
and female and male delta p-values are indicated in 
parentheses). 

'*Xhe ballot below was^used in a general election." Look 
at the ballot t6 answer the questions on this and the 
following two pages. * 



* OFFICES 


LEGISLATIVE 


COUNTY • 


SENATOR 
IN CONGRESS 
(vote for one) 


REPRESENTATIVE 
IN COr^GRESS 
(vote for one) 


COUNCILMAN 
(vote for two) 


TAX ASSESSOR^ 
(vote for one) 


DEMOCRATIC 


Alan F 
KIRK 


'• f 

John G 
SMITH 


Martha G 
DAVIS 


Peter V 
MOSS 




REPUBLICAN 


James M 
JONES 


Mary 
O'CONNOR 


John 
RICHARDS 


Michael M 
MERWIN 


Joseph L 
LASKI 



A. If you wanted to vote for Kirk for senator, could 
you also vote for O'Connor for member of the House 
of Representatives? 
0(Nation4%72.8) Yes 
(Female -V.34; Male 1.41) No 

I don't know 



B. Could yfou vote for both Davis and Moss for coun- 
cilman? 



(National % 83.2) Yes 
(Female 2.11; Male -2.23) No 

* I don't know 
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C. CoUlli you vote' for both Davis and Merwin for 
* councilman? 

(Na2ional%73.6) 3^ 
(Female .35; Male -.37\ No 

I don't know 

^ D. If you were registered as a menibcp of the 
Democratic Party, could you vote for Laski for tax as- 
sessor? • ' f 
" (National % 62.6) • ^ ^ 
(Female -2.78; Male 2.^3) .No. 

I don't know \ 

. E. Could you vote for both Kirk and Jones for 
senator? 

(National % 90.1) Yes . 

(Female .29; Male -.31) . tifi ' ' 

I don't know." 

Even in this set of questions whertf 1 7-year-old females 
scoreS their only significant advantage (part B), they suffer 
two disadvantages, one of which is significant (part D). On 
part D, which reteived the lowest national percentage of any 
part, the answer cannot be deciphered solely from informa- 
tion on the J;}aIlot. Other information about general elections 
must also be known. Perhaps females are a bit better in 
figuring out* the ballot^ puzzles but are less informed on elec- 
tion rules. 

At the adult leyel, the differentiation of males and females 
on the criterion of this Structure color scheme is complete. 
The male adults are. above the national percentagO'on every 
. one of the 21 exercises included; the female adults are below 
on all! AH but 3 delta p-values out of th« 42 registered for the 
two groups are significant. The lowest advantage scored for 
«fhales is 2.6%; the greatest is 

Males and females are most nearly alike in their resp9nses 
to the votingcexercises cited above and pn tv^o questions 
about the federal government. One of these is u(irelc5ised; 
the other asks^which level of governnient could raise mail 
rates. They are most different on four detail questions about 
the ffedcral govci nmcnt. Two pf these are unreleased. The 
others ask which. branch of government can declare a 
Congressional act unconstitutional and how a presidential 
candidate is nominated. , i> 

These results may suggest that the school and the society 
cooperate in the depoliticization of females in this country. 
As they grow older, n^ales display an increasing advantage 
"ovqr females on knowfedge about ^ politij:al matters and 
political structures. " 



Rights and Duties. Exhibit 18 depicts male, and female 
median delta p-values for the color scheme on Rights and 
Duties across three age levels of 13, 17 and adult. Age ?, as 
we have said before,, has only one exercise in this theme and, 
therefore^ is not includ^4here. 

The median delta p-vaiuc« f-r^r age 13 suggest that males 
and females are much alike on the 10 exercises summarized. 
tA review of the delta p-valtles for J'idividual exercises se^ms 
^ to confirm this judgment. The females have 7 but of 10 exer- 
cises above the national perceryage of success, but none 
more than 2.1% above.Since all male/female distributions I'n 
these data are inversions,* the males necessarily have seven 
exercises below thei national percentages but none more 
than -2.0%. The twoWreatest female advantages are on two 



unreleased exercises about freedom of speech and religipn 
and due process. At the other end of the distribution, the 
males register a 6.2% advantage on an exercise that aoks, 

Should a newspaper or magazine be allowed to 
publish something that criticizes an elected govern- 
ment official? ' V 
Yes 

. No . 

Undecided. 

At age 17 the median positions of the tv/o^ groups are 
reversed with males being slightly above females. The dis* 
persions are greater at age 17, however. Female advantages 
oa five exercises in which the two groups are significantly 
different range between 3:3% and 9.5%. The females are 
significantly more successful on exercises referring to race as 
a factor in empToyment and freedom to criticize military ac- 
tions and on a two-part exercise that asks, 

A. Should a congressman pay attention to the opi- 
nions and concerns of people whose views are dif- 
ferent from those of the majority? 

Yes > 

No - 

Undecided 

B. Please explain any answer you selected. 
[Written answers judged acceptable or unaccep- • 
table]. 

Males, oh'the othe^r hand, did significantly better on exer- 
cises referring to the right to picket to show dissent, 
atheists holding office and newspapers criticizing public' 
officials. 

•While the difference in median delta p-values between 
adult males and fem'ales is only about 3.5%, the variance on 
individual exercises is considerably greater than for either 
13 or 17-year-olds. For e;can\ple, males scored a 15.3% ad- 
vantage over females on the question wh,ether persons v/ho 
do not believe in God should be allowed to hold public of- 
fice. Fewer thari half the women sampled were willing to 
have arratheistic public official. The males also have about a 
12% advantage over females on questions 'whether persons 
should be allowed to picket iaprotests against rock festivals 
and alleged police ""brutality.^^ver haJf the women (57%) 
\vould allow the picketing of rock festivals, but fewer than 
''^half (46%) would allow the picketing of police stations, 
vj^hile two thirds of the men wpuld aljow picketing against 
the festival, only a slight majority would permit picketing 



*This inversion phenomenon is easily explained given, the fact 
that the male and female samples are roughly equal In number in 
all groupings and, of coursef given the necessary inclusion of 
everyone assessed in one or the other of the halves. Thus, the 
national percentage of success on an individual exercise woqld lie 
midway between the males* percentage of success and the females' 
percentage of success. Whert converted to delta p-values, a male 
+ value Would be matched by an equal female - value or vice versa 
When delta p-values are ordered in a high-to-|ow distribution, the 
inversion of scores and exercises occurs. A similiar relationship is 
evident in black and white distributions, but the difference in 
numbers in the two samples and the presence of the "other" 
catcqdVy make the inversion imperfect. 
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against repDtted police brutality. Most aclults (77.5%) 
believe that persons should be allowed to assemble in a 
public place to make demands known. Males, however, are 
. 10% more likely to support s^uch actions. 

Females, bn the other hand» post some advantages over 
males. They have about a 6.7% advantage over the men on a 
question about due process in a given situation and are 
slightly more likely than men to reject race as a factor in 
employment. Differences this small could, of course, be due 
to chance. 

It would seem en the whole that males display a greater 
knowledge -of or^reater commitment to democratic rights 
and duties than females. This male advantage emerges 
betwieen ages 13 and 17 and increases through young 
adulthood. One is tempted to conclude that females are 
socialized in this society to be less interested in, to know less 
about and possibly to be less committed to certain political 
values than malesc This conclusion seems consistent with 
common observation. The politicized female still is an 
anomoly in this society. Schooling likely contributes to the 
depoliticization process. Schools, however, are only one fac- 
tor. ■ . 

Race 

Exhibits 19 and 20 display the results of these In- 
stitutional exercises subdivided into Structures and Rights 
and Duties color schemes and analyzed .by black and white 
races. A third ^fategory of persons classified as "other'* — 
i.e., presumably neither black nor white — is excluded from 
this analysis because of the few numbers of persons in the 
category and some inconsistencies in the identification 6f its 
members. 

Structures. Exhibit 1 9 displays the median delta p-values for 
the two racial groups across the four age samples for the ex- 
ercises classiHed as Structure. 

The variance between blacks as a group and whites as a 
group on Structure-related questions is apparent in Exhibit 
19, The median delta p-values arc between 15 and 24 
percentage points apart. "While half of the individual exer- 
cise results are closer together than these medians, no in- 
dividual exercise result for the two groups is closer than 
6.6% at age 9, 2.8% at age 17 and 10.3% at the adult level. 
Only at age 13 does the black group score higher than' the 
white group on individual exercises. There the blacks have 
advantages on two exercises of 2.9% and 8.5%. At their 
greatest differences on individual exercises, whites score ad- 
vantages of 22% at age 9, 33% at age 13, 30% at age 17 and 
39% at the adult level. 

As in the previous displays of the overall results from all 
exercises in Exhibits 3, 4 and 5, the confirmed picture is that 
of two groups significantly and dramatically differentiated 
by their performances ^n these NAEP exercises. 

At age 9, the two groups approach each other tlie closest 
on a question that asks, 

In a court, which one of the following has the job of 
making sure that the trial is fair and run according 
to the rules? 

The judge 

The lawyer 

The jury 

The person on trial 
I don't know. 

Nearly thjree fourths of all 9-year-olds could answer that 
question correctly. Whites are only 1.6% above the national 



percentage while blacks are only -5.0% below. Black and 
white 9-year-olds cdme nearer equally sharing this informa- 
tion than any other in this color scheme. At the other ex- 
treme, only one third of all 9-year^olds know that heahh 
departments inspect restaurants. When divided by race, 
/-about 40% of the white children aaswercd the question cor- 
rectly, while only 17.6% of the black children knbw this fact. 
Explanations for such results tumble^ out rapidly. As a 
group, 9-year-olds are not especially aware of the ordinarily 
unsung governmental function of restaurant inspection. Just 
as likely, they are not sensitized to cuiinary cleanliness or 
pOs$ibly to restaurantss themselves. Beyond this, black 
youngsters probably frcquwit restaurants less than white 
youngsters. 

On the 6ther hand, blacks frequent courtrooms jn this 
country as trial principals far beyond what their population 
proportion would predict. Furthermore, television dramas 
offer recurrent though usually distorted views of courtroom 
scenes in which judges mainly play referees between wily 
lawyers. • ' 

In another instance, over 60% of the white 9-year-olds 
' could identify the mayor as the head of a town government, 
while only 45% of the black 9-ycar-olds could do so. 
Cultural explanations of this 15% variance are harder to 
come by, but one possibility is that blacks in this country 
tend to be either city or farm folk. Towns are predominantly 
white. Thus, probably a smaller proportion of black than 
white 9-year-olds in the sample would be town residents. 
However, we do not have that kind of information on the 
composition of these samples. 

At age 13^ the Structure color scheme is assessed by 
means of 16 exercises. The range of successes over these ex- 
ercises for white 13-year-olds- goes from -1.0% below the 
national percentage to 4.5% above; the range for black 13-^ 
year-olds goes from -29.0% below the national perc<^ntage ot 
7.5% above. 1/ 

On one unreleased exercise for age 1 3 concerning a func- 
tion of a major department of the federal executive, black 
13-ycar-olds achieved an advantage of 8.5% over the whites. 
It is also the exercise on which black 17-year-olds did best 
while white 17-year-olds did very poorly. In both cases, 
since the whites, who outnumber the blacks in the samples 
by 6 and 6.5 to 1, did badly, the national percentages of suc- 
cess arc quite low. Only 27% of all 13-ycar-olds and 38% of 
all 17-year-olds answered .the exercise correctly. The result 
may have very little meaning, actually, since the national 
percentage is at the chance level of performance at the 13- 
year-old level. 

On three related questions about government services, 
black and white 13-year-olds arc extremely divergent in 
their successes. Students were asked "Which level of govern- 
ment (federal, state or local) would be most likely to pass" 
the following: 

... an act to raise the rates for sending letters 
through the mail? 

... an act to lower taxes on goods coming into the 
country? 

... an act to increase garbage collection services? 

The national percentages on the three questions are 71.7%, 
73.4% and 77.0%. Tne white 13-year-olds arc above the 
national percentage in each case by 3.6% to 4.5%, but the 
blacks register deficits from -2f.0% to -23.9%. A bare ma- 
jority of black 13-year-olds could answer the questions cor- 
rectly, while over three quarters of the whites responded cor- 
rectly. 
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A whole sei of cultural eflects is probably operating here. 
It would seem gratuitous to point them out. Whether they 
are sufficient to explain the 25% disadvantage of blacks is, of 
course^ the central problem. 

We^ili mention one other exercise, why^is the locus of 
poor performances of all the samples in the assessment. Age 
13 respondents scored their lowest national percentage of 
success on this exercise. They were asked, 

The presidential candjcjiate for each major political 
party is formally nominated by, which one of the 
following? 

The Senate 
* A national primary 
A national convention 
The House of Representatives. 

Only 17.0% of the 13-year-olds could answer the question 
correctly. Whites managed to make that 17.9%; blacks 
scored a bare 10.8%. Presidential campaigns seem not tpbe 
. a major interest of American 13-year-olds students, black or 
white. 

At age 17 the median advantage of whites ovejhblacks in- 
creases to 19%. This difference reflects a range of differences 
on individual questions from as little as 2.9% to as great as 
32.1%. All of these differences in dejta p-values are white 17- 
year-oid advantages pver black 17-year-olds. Thus, on all 
but one exercise whites scored significantly above the 
national percentages; on all exercises black 17-year-olds 
scored below the national percentages, an^Tall but one are 
^ a significantly below. The black and white 17-year-olds 
V f. ' : ^fsponded alike — i.e., neither is significantly different than 
the national percentage for all 17-year-olds — on an un- 
released exercise about the functions of a department in thp 
federal executive. Albeit, that result is by far their lowest 
national percentage of success; only 37.7% of all 17-year- 
olds marked it correctly. 

Black and white 17-year-Qjds ^e most different in their 
responses to two questions, about the federal government 
and one question about a function of local governments. On 
the two of these that are unreleased, one of the federal 
government questions and the local gpv^rnment question, 
53% and 88% of the white 17-year-^s marked.then{ cor- 
rectly while only 28% and 57% of the black 17.year-olds did 
so. The released federal question asks how a presidential 
candidate is nominatec*. While barely a majority of whiter / 
could answer this question, only one in five of the blacks 
marked it correctly, less than could be expected on the basis 
of guessing. 

Among adults the differences between blacks and whites 
widen still further. While there is general gain in adult 
national percentages of success as well as in all adult raw p- 
values for both blacks and whites ovenjcomparable values for 
17-year-olds on these Structure questions, all of which are 
shared by adults and H-year-olds, the blacks tend to gain 
fewer percentage points and less consistently than vhites. . 
All white adult delta p-values are significantly above the 
national percentages and all black adult delta' p-values are 
significantly below the national percentages. Within their 
ranges of typicality, black and white scores on individual 
questions vary from 19 to 30 percentage points, always with 
a white advantage. Where they approach each other in per- 
formance — i.e., on an item where blacks do their atypically 
best and whites th^ir atypically worst — the whites post 
only a 10.4% advantage. This unreleased exercise has to do 
with the armed forces. At the other end of the scale, on the 
question about national political conventions, whites have a 



39.2% advantage. Only 26.2% of the black adults could 
^ns^er the question, a guessing score. Nearly two thirds of 
the white adults know that presidential candidate^ are 
npmXnated by national conventions. 

ThrWe of the four questions that arc the loci of atypical 
f perfo^ances by both blacks and whites are also among the 
fivG^asiest questions for adults. The national percentage for 
L each of the five exercises is above 90%. On one of these, the 
ballot question where respondents are asked if they could 
voie for both Democrat Kirk and Republican Jones for the 
single Senate seat, 80% of the* blacks and 92% of the whites 
could answer correctly. On another, 97% of the whites and 
82% of the blacks noted that the federal government could 
raise mail rates. • - ' -* 

On two other questions, blacks do comparatively worse. 
Only 71% of the black aduhs as compared to 95% of white 
adults could relate federal government with the control 
of tariff rates. At the local level, only two tJSirds of the black 
aduks connect garbage collection with local government; 
95% of the white adults know this common local government 
function. 

While there is improvement in the performances of both 
blocks and whites from the lower to the higher age samples 
on the shared questions, it seems undeniable ^hat the 
relative disadvantage of the black samples increases with in- 
creasing age. Schooling may contribute to the ability of all 
persons to respond to questions such as these, but it is not 
effective in reducing the performance gap between these two 
racial groups in our country. That Such a gap can be 
described racially is evidenced in this NAfiP assessment and 
in most other evaluati^ efforts that enjoy mainstream sanc- 
tions. That such gaps become defined racially is the unfor- 
tunate and unwarranted consequence. 

Rights and Duties. Exhibit 20 displays the group effects of , 
the three black and white age levels for which there are exer- 
cises cla'ssified as measuring a qiix of knowledge of and com- 
mitment to some Rights and Duties associated with some of 
our social institutions. . ' 

The relative advantage of white respondents over black 
respondents ranges from 8.5% at age 17 to 17.9% at the 
adult level. The apparent gain in black performances noted 
at age 17 on this set of exercises is also reflected in the 
Southeast group on the regional c^isplays and in the low 
metro group in the STOC exhibits. The Southeast and the 
inner city are likely locales of the majority of blatk respon- 
dents in these samples. 

Within their ranges of typicality, black and white 13-year- 
olds differ fronr v :l • ^ther from 7.2% to 17.4% on individual 
exercises. Witl.n. this group of questions to v/hich both 
groups responded typically is the question. 

Should ractf be a factor in hiring someone for a 
job? 

Yes 
No 

I do'^\ know. 

"No" is the response of 90.5% of the white 13-year-olds arid 
the response of 83.3% of the black 13-year-olds. 

The two groups are actually closer together and unex- 
pectedly more settled on a question that asks. 

Do you think there should be laws against acts of 
^-vandalism such as destroying a statue? 
Yes 
No 

Undecided. 

Neariy 95% of the blacks and 93% of the whites approve of 
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such laws. This is the only question on^ which black l3-year- 
olds post an advantage over whites. A contrast in perfor- 
mance and in content is found in an unreleased exercise 
referring to religious freedom. While 81% of the whites sup* 
^ ported this value, only a bare majority of blacks did. To sug- 
gest that thp responses to the two questions, especially 
among blacks, demonstiate a disparity between human and 
property values is probaUly too facile.^ 
) At age 17, the Rights and Duties color scheme is defined 

by 21 exercises. Nine questions arc shared with the 13-year-, 
old group and all but one are shared with the adult group. 
Asi. in all Other distributions featuring black and white 
respondents, the dispersion of the whites' delta p-values is 
quite compact, running from -1.5% to 4.8% for a total range 
of only 6.3 percentage points, while the dispersion of the 
blacks* delta p-values is broad, running from -24.5% to 7.6% 
fQr a total range of 32.1 percentage points. 

The two groups are practically alike in their responses to 
* the question involving racial discrimination in eiriplpyment 
and an unreleased question on age discrimination. On the 
racial discrimination exercise, the two groups' performances 
are within 0.8%; on the age discrimination question only 1.4 
perceijtage points separate the two groups* results; The 
national percentages for the questions are 93.8% and 89.3%. 

Black and white 17-ycar-olds are most different on a ques* 
tion involving the freedoms of speech and refigion and a 
question that asks. 

Should a person who does not believe in God be al- 
lowed to hold a public office? 
Yes ^ ^ 

No .■ 
Undecided, 

The groups' responses vary 21.2% on the firsthand 29.1% oh 
the second. About 60% of the black 17ryear-plds appear to 
be supportive of religious and press freedoms jn the first, but 
^ only about 38% are supportive of religious freedom in the se- 
cond, as compared to 82% and 67% of the white 17-year- 
oids. 

Also among the most varied responses are those related to 
the question. 

Do you think people should be allowed to picket 
the holding of a rock festival as a protest against It? 
Yes 
No 

* Undecided 
No response. 

A small majority of 52.7% of all 17-year-c>lds would support 
this sort of action. As a group, whites are slightly more will- 
ing; 56.5% would approve. However, among black 17-year- 
olds, only 32.8% are approving. The contrast between 
blacks and whites is complicated by the fact that just half 
the white 17-y^r-olds approve of the picketing of a police 
station to protesXreported police brutality; but 59.2% of the 
blacks would approve. Roughly speaking, almost twice as 
many black 17-year-olds would allow a picket protest 
^ aE^ainst alleged police brutality as would allow a picket 
protest against a rock festival. The effect 6f social experience 
on the way persons respond to presumably similar situa- 
tions, picketing, in this case, seems well demonstrated. 
Perhaps, the effect of social experience on the way persons 
respond to NAEP test questions is better demonstrated. 
Adult black and whiite respondents demonstrate the 
, same patterns on Rights and Duties exercises as do the 13- 
^ year-old and 1 7-year-old samples. All but two delta p-values 
of whites are above the national percentages and all but two 



ddta p*values of blacks are below the national percentages. 
The median delta p-values in Exhibite 20 suggest further 
that on Rights and Duties exerciies, as was also noted 
earlier for the Structure excrciscs,'biack and white adults 
differ more from each other than do either the 13 or 17-year- 
old blacks and whites. 

On three exercises in this color scheme, black and white 
adults are most alike, i.e., neither group is significantly dif- 
ferent than the national pexccntagcs of success. It is on these 
exercises that the Whites produce their lowest delta p-»values 
and the blacks produce their highest. The two released 
. questions refer to the factor of race in employment and 
picketing a police station. Over 90% of both groups believe 
race should not be a factor in employment and ju$t over 50% 
of each group believe that persons should be allowed to 
picket against brutality. The radicals among the adults are 
those persons classified as '*other*' who arc not included in 
this report. They posted an impressive advantage of 16.9% 
on the police-picketing exercise 

The adult groups are most different on three released ex- 
ercises where the whites are from 3.7% to 5.0% above the 
national percentages and the blacks- are from 20.4% to 
29.9% below. The three exercises, previously referred to in 
other sections of this chapter, concern atheists holding 
public office, picketing a rock festival and the Supreme 
Court decision on religion in public schools. The national 
percentages on these questions arc 55.6%, 63% and 52.3%.. 
The principles involved in these questions are either not too 
well known or are not too well regarded by this sample of 
adults or are opposed, by other values unknown. 

According to these results^ the principles are also not 
equally known or regarded by these black and white adults. 
In the following exercise, 57.4% of the white adults sampled 
recognize the principle of separation of church and state, 
but. only 22.4% of the sampled black adults know the 
grounds for the famous decision. 

The Supreme Court \ruled that it is uncon- 
stitutional to require prayer and kbrmal Religious 
instruction in public schools. 
Which one of the following was the basis for its 
decision? 

The requirements violated the right 
to freedom of speech. 

There was strong pressure put on the 
Supreme Court by certain religious 
minorities. 

Religious exercises violated the principles 
of the separation of church and state. 

■ \ 

Every moment of the valuable school time 
was needed to prepare students to earn 
a living. . * 

I don't know. 

"I don't know" presumably functions differendy than a sub- 
stantive distractor in such a question; however, given the 
right assumptions about the four distractors in the exercise, 
guessing by the entire black adult population in this sample 
could be expected to yield the same result they produced. 
Neither do the whites earn accolades for their performance! 
Chapin's team, in another inquiry into this assessment, es- 
timated that at least 60% and possibly more than 80% of the 
adults should be able to answer this question. 

Besides the chance results of guessing, conjectures- as to 
why these adults performed so pooriy Here are myriad. They 
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would have to include, would Seem, the possibiHty that 
these adults, in the absence bf^call knowledge about the 
casc^' respond more readily to suggestions of political prac- 
ticality than to moral principle and, therefore, project the 
same posture to others, including Supreme Court justices. It 
is far more practical to suppose that the court yielded to 
pressure — not a bad guess, really, given the political nature 
of that institution. 

In another of these differentiating exercises, about 60% of * 
the white adults appear willing for a person who does not 
believe in God to hold public office. That majority is not 
likely to get an atheist elected in many Communities, 
however, for only about 35% of the black aduhs are so in* 
clined. The anomoly is that blacks, who have been extraor- 
dinarily influential in expanding civil rights relative to race,^ 
appear not to extend these principles to other realms such as 
religious belief. 

The last of these exercises also suggests some contrasts. 
Some 67% of the white adults would allow pickets to protest 
a rock festival. Only 39% of the black adults would do so. 
The contrast between the groups anent rock festivals, which 
may reflect different musical tastes, is less telling than the 
contrast with these groups* previously cited willingness to 
picket against reported police brutality. Just over half of 
both groups would allow anti-police pickets. Thus, to draw 
the lines of perspective, the blacks are more willing to allow 
active protesting of police brutality than they arc to allow 
active protesting of rock festivals. Whites, on the other hand, 
seem more concerned about the festivals than about the 
brutality. Perhaps the difference has something to do with 
whose ox is gored. 

Parental Education 

In this section^ the results for the four Hgc levels in the two i 
color schemes are analyzed across four levels of parental 
education. The median delta p-values used to characteijize 
the different groups are shown in Exhibits 21 and 22, 
x^truciures. Exhibit 2rdisplays the results in the Structures 
theme for ages 9, 13, 17 and adult classified according to 
whether neither of the parents of the respondents went! to 
school beyond the eighth grade, at least one parent started 
but did not finish high school, at least one parent graduated 
from high school but did not go beyond that, or at least one 
wettt^ on for some kind of training after graduating from high * 
school. The two exhibits attempt tc show the relative 
refationships between the respondents' answers to these 
qt^stions and the varied educational exposures of their 
parents. 

The displays^ strongly suggest that the four age samples 
can be divided into discrete populations according to the dif- 
ferent levels of parental education. The relative positions of 
th,e median delta p-values are hierarchical and consistent. 
The higher the educational attainment of the parents, the 
better the respondents perform on the set of questions clas- 
sified by this theme for each age level. 

At age 9, the sample is almost dichotomous with the no \ 
hii^h school and some, high school groups producing all 
resuhs on u dividual questions below the national percen- 
tages; 8 of the two groups* IG delta p-values are significantly 
below. \n contrast, all of the 10 deha p-values of the 
{graduated high school and post high schopL groups are 
above the national percentages; 8 are significantly above. 

Thd four^age levels come nearest performing as similar 
* . populations on the question that asks who is responsible f^r 
a fair trial. Only the post high school group answered this 



question at a level that is significantly above the national 
percentage of 73.8%. The post high school success of??. 6% 
has an advantage of 10.7% over the lowest no high school 
group. ^ 

The post high school group's advantage over the no high 
school group is 18.0% on the "town mayor** question and 
16.5% on the "restaurant inspection** question. On both of 
these questions the post high school group is significantly 
above the national percentages while the nO high school" 
group is significantly below. Only 44% of the no high school 
group know that the mayor is the head of a town govern- 
ment and only 27.2% know that heahh departments inspect 
restaurants. Few 9-year-olds in any category know this 
health information. Only 44% of the post high school group 
marked the question correctly. 

The dichotomy between the post high school and 
graduated high'school groups and the some and no high 
school groups observed at the 9-year-old level seems not to 
hold at age 13. For one thing, though not detectable in the 
median values shown in jBxhibit 21> three of the groups* 
Scores are far more disperse at age 13 than at age 9. At age 9, 
the spreads of the four age categories on five observations 
are 5.9%r, 8.6%, 4.1% and 4.4%. At age 13, the spreads on 16 
observations ^re 26.5%, 15.1%, 4.2% and 13.4%. The 
anomoly is the graduated high school group with a disper- 
^sionofonly about 4.0% in both age samples. There seems no 
self-evident explanf tion for the anomoly. The graduated 
hicfh school group simply performed close to the national 
„ percentages on all the exercise^*. The som?,'and no high 
school groups hover near the national percentages on 
perhaps one tjiird of the exercises but plunge 16% to 2*^% 
below on others — though not necessarily on the same one's. 
Meanwhile, the post high school group soars above the 
national percen^ges from 2.4% to 13.1% on all but one exer- 
cise. 

Tne groups performed alike only when everyone appears 
to hsive been guessing. For example, an unreleased exercise 
asks for the identification of a federal executive department 
by its major function. All scores converge to the national 
"guessing" percentage of 26.7%. 

The greatest differences in the groups are registered for 
the two exercises that ask for the identification of the level of 
governmenf — i.e., federal, state or local — most likely in- 
volved in lowering taxes on incoming goods and in increas- 
ing garbage collection. The post high school group earned 
its greatest advantages on these questions with successes of 
13.1% and 11.0.% above the national percentages while the 
no high school group, in contrast, suffered its greatest disad- 
vantages with successes of •-16.fe% and -23.7%. The 
gradugited high school group, which deviates from the 
national percentage but little on any question, has a deha p- 
value of -3.1J5% on one and -2.35% on the other. The some 
high school group has one of its worst scores on one, -11.9%, 
and one of its best on the other, -1.3%. 

Generalizations at the 13-year-old level, other^an those 
suggested above, are elusive. While overall the 13-year-olds 
do not perform well on this set of questions, in general they 
do somewhat better on exercises whose contents have recur- 
rent reinforcement in day-to-day experience. For example, 
practically none know how a presidential candidate' is 
nominated (17% for all; from 12% to 20% by group); but 
most know how men get into the armed forces in wartime 
(82% for all; from 70% to*90% by group). There are resuhs, 
however, that make this generalization less than .firm. For 
example, more no high school respondents know that a 
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scna IT usually is elected rather than appointed (70%) tha« 
know that local governments collect garbage (52%). 

. At age 17, the groups appear to perform much- as they do 
at age 13. All post high school IT-year-oIds* delta p-values 
are above the national percentages and all but one are 
significant; all no high school delta p-Values are below the 
national percentages and all but one are significantly below. 

• The some high school 17-year-oids look a bit like a replica of 
the iio high school group, but withwdelta p*values elevate<J 

" about 5 percentage points. Nonetheless, all but one of these 
values ai*e below the national percentages; all those below 
are significant. The 5% differtnce i$ accui^ately reflected in 
t^ median delta p-values in Exhibit 21. ^The post high 
school 17-year-olds are something of a reversed image of the 
some and no high school groups with their best relative per- 
formances on exercises on which the other groups performed 
most poorly. The dispersions of these thiTe distributions are 
also quite large. The no, some and pos^4^igh school groups' 
spreads, in that order, are 20.7%, 22.6% and 11.3%. The 
graduated high school group maintains its anomolous 
character. Its delta p-values are all within about db3% of the 
national percentages^ half above and half below. The list of 
21 exercises, ordered from the group's best to poorest perfor- 
mances, docs not resemble the lists of the other groups when 
ordered by the same rubric. An acceptable explanation is no 
more apparent for ^ the 17-year-olds than for Jhe 13-year- 
olds. 

One explanation, which lies somewhere between a pure 
conjecture and a random' guess, is that the group who^e 
parents graduated frdm high school but never went beyond 
.high siphool has the Strongest orientation to school. It 
becomes the Ihprming; group on these exercises, which also 
reflect^a strong taint of schooling^ The other groups, those 
whose parents were less or mo*"^ successful" in school^ deviate 
'] in different wjfys froni the norm. , 

There seems to be some evidence for this hypothesis. The 
best example of it relates to the exercise on the nominatibn 
of presidential candidates by national political cqnventions. 
The national percentage for this question for all l7-year- 
olds is 48.5%. The graduated high school group attained 
48.6% on this question." Both the some ^ rid no high school 
groups turned in their worst perf6rmances on this question 
with delta p-values of -18.8% and -22.6.%, respectively, while 
the post high school group turned in i^best performance 
with a delta p-value of 12.1%. 

In .another instance on an unreleased exercise concerning 
school governance, the graduated h,igh school group is about 
1.0% below the national percentage and within 0,9% of its 
own oiediah delta p-v'alue for all of thest exercises, while the 
no antl some high school groups* produced their best delta p" 
valuc^ of -1.9% and 3.8%, respectively, and the post high 
' schodT group produced its worst delta p-value of 0.7%. 

At the adult level, differences in the groups continue to 
manifest themselves. Inequalities in pcrfomance in this as- 
sessment associated with different family environments with 
respect to parental education are not erased by time. Having 
been reared by parents who did not attend high school ap- 
pears to disadvantage adults with respect to this assessment, 
in much the same manner that it appears to disaclyantage 
others of similar backgrounds of school age. Furthermore, 
while there is some reason to infer from these data that there 
is a slight improvement in test performances of this kind by 
persons after 1 7 years of ag^, there is more bas»s for inferring 
that the important gains ar^ made during the school ages of 
13 and 17; These inferences can be made largely from the 



median national raw p-values shown in'Exhibits 3, 4 and 5 
in the earlv part of this chapter. The evidence here relative 
to Structure information suggests that among groups all 
such improvements are parallel. No group — here, for ex- 
ample, groups categorized by parental education — escapes 
its past. _ - 

An 'example of the kind' of improvement referred to is in 
the question about a presidential candidate's nomination. 
Less than half of all 17-year-olds could answer this question, 
but nearly 60% of ^11 adults, know this function of national 
c^ventions. That would count as improvement if it is more # 
than a chance gain. The persiAcnt inequalities are shown by 
the* fact that as adults, still less than half (44%) of the 
persons whose parents never went beyond eighth grade 
know how presidents are nominated. About 57^0 of the Some 
high school group know this fact along with about 64% of 
the graduated high school. In contrast, 80% of the group 
whose parents went on to school after high school marked 
the question right. 

Some things, it seems, nearly all adults know. That the 
federal government carries the mail is known by 95% of all 
aduhs. Even here, however, group diffei-ences persist. The 
no high school group posts a high 92.3% on this question, 
but the post high school group hits a crocking 99.2%. 

A phenomenon of these aduU dMk when car<2gori2ed by 
parental educ?>tion i3 the uncannily accurate ordering of the 
exercises by difficulty in the pD§t high school distribution. 
That is, on the hardest exercises, those which received the 
lowest national percentages of success, the [ bst high school 
group achieved its highest deltn p-scoves. But on the v.asiest ^ 
questions, where the national percentages of success were 
highest, the post high school group, achieved its lowest delta 
p-values, 

CDnc's first impression is that this group did best on the 
hard questions and poorest on the easy questions; bu' that is 
not the case, for on the easiest question of aU for the adults 
as a whole, the post high school group has a percentage of 
success of 99.3%^ That mcdni only three persons \n Xhc 
whole population missed the question. The fact ^s that the 
• post high school group did well on all the queshons. When 
the national percentage was high, th^re was uimp^y less 
range on top f^r the post. high school grptip*s advantage. 

A second hypothesis h hazarded. The most consistent, ef- 
fect ivetind long-lasting infiuencc-on the ability to respond to 
questions of this kind is the educational environment of the 
home. Another way to put it is that parents who are succes- 
sful in school have children who are successful in school. 
That assumes, of course, that the ability to answer these 
questions is related to or is more generally expressed as the 
ability to succeed in schooL Persons who are successful in 
schooiwork are successful partially, at kast, becav^se they 
•develop interests in the things schools stand for anc^ 
emphasize. To some e?^tcnt. perhaps to a great extent, such 
school- related interests are retained into adult life and are 
perpetuated as interests in the liv^s of children, *'visitlng," ' 
as it were, **the iniquity of the fathers upon the children to 
the third and the fourtli generation 

Rights and Duties. Exliibit 22 displays the median deka p- 
vaiues for three age levels for thic exercises in the Riglits and 
Duties theme as categorized by parental education, Age 9 is 
not represented because of too few exercises in this theme. 

In genera! terms, the relationship of parental educiaion to 
percentages of success in these thematic arrays is consistent 
wkh what was found in the Structure thematic distribu- 
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tions. Sorting on the fourievels of parental education ap- 
pears to result in four distinct populations that perform con* 
sistently in |his theme at all age levels. The post high school 
group scores advantages consistently; the other groups con- 
sistently score relatively lower successes. 

As we have (Emphasized before in reviewing this theme, 
responses in this* theme may carry a heavier value burden 
than those in the Structure theme. Knowledge of Structure 
questions arc based largely on factual recall while the ques- 
tions classified a« 'Knowledge of Rights atid Duties are an ^ 
uncertain mix of fact and value. To some degree, the delta p- 
val.ues reported in this theme reflect a rating and ranking of 
preferences associated with our democratic institutions. In 
other- respects, they reflect a knowledge of the principles and 
rules by which our institutions are governed. To say, for ex- 
ample, that race should not be a factor in employment 'may 
indicate onb's preference in the matter or it may indicate a 
knowledge of the law of the land, which explicitly prohibits 
racial discrimination in certain jobs and implicitly dis- 
courages it in most otheVs. 

Among 13-year-olds, a distribution pat'tern similar to that 
noted earlier for Structure is evident. Dispersions of the no, 
some and post high school groups are fairly wide — 21, 14 ' 
and 1 1 percentage points, respefctively — while the dispei^ 
sion of the graduated high school group is narrow, only 5.5 
perdfentage points. Furthermore, all the no high school delta 
p-values are below. the national percentages, as are all but 
one of the some high school group's scores. In contrast, all 
post high school values are above the nationa] percentages. 
The graduated high school group's delta p-values are half 
above and half bHow. 

On only one question did all groups respond in the same 
way; 93% of ail groups, ±0.4%, agree that ther<; should be 
^aws against vandalism. On all other e:jcercises there is 
significant variance by one or more of the groups, r^o other 
question received so high a national percentage of success. 
sense of property rights still appears to be alive and well}% 
least among these 13-year-olds. % 

At the opposite end of the success Scale, only 48.9% of all 
l,Vyear-olds responded affirmatively to the following ques- 
tion: 

Should a newspaper or magazine.be allm^^ScTlo 
publish something that criticizes an eleaed 
government official? 

Yes ' . 

No . , ' ^ 

Undecided. 

This question also pEoduced considerable variance among 
the groups. The pos! high school group is 10.5% above the 
national level on the question" while the graduated high 
school groiip, its nearest competitor, is -3.9% below. The 
some and no high school groups are far below the national 
percentage at -14.5% and -13.1%. While 9 out of 10 of the 
respondents whose parents either did not go or did not 
graduate from high school believe that vandals should be 
punished, only one out of three believe government officials 
should be publicly criticized in the media. In addition, fewer 
than half of these groups would allow an atheist to hold a 
public office. On this question, over 60% of the graduated 
high school group and 67% of the post high school group 
would permit a person with such a belief to Jiold office. 

One is tempted with such resjults to apply labels of liberal 
and conservative\to the groups. However, conservative is 
hardly the proper appellation for a group two thirds of 
which would not allow, newspapers to criticize the govern- 
ment; nor is liberal an especially appropriate way to refer to 



a group one third which would apply a religious criterion to 
public , service. The application of a value-conflict nioclel to 
these data would likely surface a number of both inter- and 
intragroup conflicts that warrant some attention. 

No great liberal group shows up at age 17 either; though a 
tendency in that direction can be inferred for the post high 
school group on some questions, if certain assumptions are. 
made about the?tnfeaning of liberal, the meaning of the ques- 
tions and the nUraning of the responses to the questions. For 
example, the post high school 17ye<U'-olds have significantly • 
positive responses of 4% to 10% above the national percen- 
tages on questions involving public assembly, picketing 
police stations and rock festivals, criti'cizing'government of- 
ficials and policies and atheists holding public office. Their 
advantage over the some and no high school groups on these 
questions runs from' 5% to 30%. If the assumptions can be 
made that a person supporting the activities listed above can 
* properly be thought of as more liberal than a person not 
supporting them and that support or approval is what is be- 
ing indicated by the positive marking of those questions, 
then it follows that the post high school group shows more of 
a tendency toward liberality than any ocher group. Admit- 
tedly, the conclusion is tenuous. It would be just as valid to 
suppose that the exercises measure an understanding of our 
way of life inasmuch as all the activities referred to in the 
questions are in accord with the law of the land. That the 
17-year-olds achieve on these exercises national percentages 
of success of only 80% to 90% may more aptly suggest an 
ii.iperfect acquaintance with some elements of our 
democratic system. 

There is evidence in two other sets of exercises in. this 
theme that suggests that the shared orientation of this 17- 
year-old samplers "law and order." The inference is derived , 
from a set of questions on which there is fairly high agree- 
ment among these respondents and little variance aijiofig 
the groups in comparison with a set of questions on which 
tfiere is relative low agreement and large variation across 
groups. ^ 

When asked if there should be laws against vandalism 
and if a crime against property would be reported by them, 
92.6% and 74.4% of these 17-year-olds answered affir- 
matively. There is nothing in those responses that is neces- 
sarily improper, of course. Percentages that high simply 
suggest high concern 'and strong agreement. No group 
answered either question in a significantly different way. 
' In something of a contrast, when asked if they would al- 
low picketing against a rock festival and picketing 'against 
reported police brutality, only 52.6% and 51.6% of these 17- 
year-olds answered^ affirmatively. FurtJiertnore, when 
analyzed by the four education categories, ^rfrere is cbn- 
.siderable disparity across the groups. Th^ no high school 
group is significantly below the national percentages by 
-19.1i% and -13/^%; the some high school group is 
significantly bclovibthe national percentages by -14.9% and 
-l.H%; the graduated' high school group is below the 
national percentages by -3.9% and -2.8%; and the post high 
school group ij> significantly above the national percentages 
by 10.8% and 4.6%. 

The point is that the 17-year-olds are/very certain and 
very homogeneous on the former questions characterfzed 
here as having a^law and order orientation; but they are 
pusilanimous and disparate on the latter questions invplving 
matters of a different order. • 

' .'\t the adult level, this law and order characterization can „ 
also be supported reasonably well by Comparing the adult 
groups* responses on the three questions that are shared by 
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the two age samples. When asked the question invc^lving the 
reporting of a crime against property, 92,4% of the adults 
answered aflirmatively, a much higher percehtage than for 
1 7-year-olds. The graduated high school adult group is 
significantly above the national percentage on the question 
by 2. 1 %. No other group answered the question significantly 
different from the national percentage. The adults, like the 
17-year-olds, are quite-certain and fairly homogeneous on 
*the question. . , 

When asked about picketing rock festivals and police sta- 
tions, 63.0% and 52.2% of the adults answered affirmatively. 
They are '10% more willing than the 17.year-olds in the one 
case and about the same as the teen-agers in the other. 
When analyzed by education categories, the adult groups, 
like the 17-year-olds, are markedly different. The no high 
school group is significantly below the natiopal^ercentages 
b'y -11.1% and -8.1%; ttie some high school group is below ' 
the national percentages by -4.3% and -4 2%; the graduated 
high school group is above the national percentages by 5.9% 
and 2.3%; and the post high school group is significantly 

^bove the national percentages by 15.2% and 12.3%. 

• Beyond this possible orientation, ther6 are other useful 
comparisons to make among the adult groups. They are also 
agreed that rate is not a proper factor in employment. 
.Adults are verHally accep^ng of -racial equality in employ- 
ment at the 92.3% level. Tliere are no differences according 
to parental education. 

At the other extreme, only 55.6% of all adults would per- 
mit an atheist to hold public office; bvit they vary greatly ac- 
cording to"^^ the educational environment of their youth 
homes. Only 41% of the adults whose parents never went to 
high scKooI would allow a nonbeliever to hold office while 
76% of those whose parents went beyond high school would 
dc so. 

Also, a bare majority of adults agree that a constitutional 
principle guided the Supreme Courtis ruling on religious in- 
struction and prayer in public schools. The post hi^ school , 
group registers a respectable 72.4% on this question, but 
or\ly 39% of the no high ♦$chooV group agree with this^ 
response. 

Size arid Type of Community (STOC) 

Exhibits 23 and 24 display the median delta p-values for 
the two color schemes. Knowledge of Institutional Struc- 
tures and Knowledge of Institutional 'Rights and Duties, 
categorized by size and type of community for the four age 
samples, 9^ 13, 17 and adult. While all seven STOC 
categories are exhibited, most of the discussions wilL refer 
only to the high metro, low metro, urban fringe and extreme 
rural groups. The justification for this delimitation is in the 
contrast afforded by these group's and in their characteriza- 
tions. The high metro group is mainly afiluent^ the low 
"metro is largely inngr city, the urban fringe is suburban and 
the extreme rural is^jirimarilj^rm. More complete defini- 
: tions are included in Appendix B. 

' Stuiciures. Exhibit ^3 shows the median delta p-values for 
these groups and age samples for the Institutional Struc- 
tures color scheme. 1 ' • ^ 

Among the varioui .STOC distributions in the 9-year-old 
sample, the most-glaring conclusion is the g^oss difference 
between the delta p-values of the hrgh and low metro 
groups. That difference, of course, is patently evident in the 
display of medians in Exhibit 23. Actually the high metro 
group differs less from the rest of the sample than does the 
low metro group. The high metro group is significantly 



' above the national percentages on only two of the five exer- 
cises in the therrie while the low metra group is significantly 
below the national percisntages on four of the five exercises. 

The lowest low metro dispersions of about -16% are On an 
unreleased questioii about ta\es and the question abcut 
restaurant inspection. On the tax question the national 
percentage is 83.2%. The high metro group is a significant 
5% above this level. None of the other five groups vary 
significantly on this questioi). The poorest performance by 
all 9-year-olds is on the restaurant inspection question. 
Only 36% know that this task is a function of Iwalth depart- 
-ments, but fewer than one out of five of the low metro group 
know this fact. That is probably below the guessing level. 

About three fourths of the 9-year-olds know that judges 
are responsible for fair trials. This knowledge is shared 
across all these STOC groups about equally. Only the 
. urban fringe group registers a delta p-value significantly 
above the national percentage; and that advantage is only 
3.6%. The high metro group is actually 1%* higher, but the 
statistical controls do not label that difference significant 
for the high metros. ^ ^ 

At age 13 there is an apparent slight narrowing of the dif- 
ferences between the high metro group and the low metro 
group as shown in Exhibit 23; but a sharper look at the dis- 
•tributions from which these medians are drawn do not bear 
this out. If anything, the two groups are significantly dif- 
ferent in more areas; but, of course, there are more exercises 
at this age level. The other groups, possibly excepting the 
urban fringe and main big city groups, are so nearly alike to 
be indistinguishable; that is, none vary essentially from the 
national percentages. 

The high and low metro groups share three exercises on 
which neither is significantly different thari the national 
percentages. These also happen to be exercises concerning 
the federal government for which the national percentages of 
success arc law, 2(iJ%', 34.8% and 47.2%. Th<:y also share 
four exercises on which they are most varied from each other 
and from the national percentages. The national percen- 
tages are relatively high in these cases ranging^from 67% to 
82.5%, Two of the questions arc about the federal goyern* 
ment, and two are aboyt local government functions. On 
. two of these four, 81 .7% and 87.4% of the high metro respon- 
dents indicated that the federal government carries mail and 
local governments carry garbage. On|y 57.5% and 61.7% of' 
the low metro students know these rc^lationships. Main big 
city respondents are not much better; 65.8% know that gar- 
bage collection is most commonly a local government func- 
tion. It is on that question that main big city respondents 
are most distinguished from other 13-year-olds excepting 
their neighbors in the inner city. Main big city and low 
metro responses to this question, which refers specifically to 
the likelihood of increasing garbage Collections, may be 
rr.ore valid comments on city sanitary departments than on 
the knowledge levels of the respondents. 

At age 17, all STOC groups except high and low metro, 
are clustered close to the national percentages of success. 
Perhaps a dozen delta p-values for the middle five groups 
are large enough ii be significant. In contrast;, ajl but five 
delta p-values for the low metro groups are significantly 
below the national percentages and all but two values for the 
high metro groups are significantly above the national 
percentages. For the most part, the two groups respond dif- 
ferently to the questions in this color scheme. As groups, 
they not only differ from each other but also from all other 
groups in the sample. 
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On two questions, however, these two groups are alike. In 
Jact, with the exception of the main big city group on one of 
these questions, the entire sample is homogeneous. On the 
ballot question, which asks if a voter can vote for both Kirk 
and Jones for a single Senate seat, 90% of the 17-year-olds 
said *'no/* On an unreleased question about school gover- 
nance, about 65% of all 17-year-olds demonstrate that they 
know the locus of a certain decision. OnJy the main big city 
group is a significant 8.4% below the national percentage. 

Three other released questions are illustrative of the dif- 
ference bet^«^n the^high and low metro groups. Two of 
these have to do w^ith^the federal government. On a question 
about raising mail rates, 93.3% of the high metro group cor- 
, rectly identified this to be a federal government function; 
only 78% of the low metro group make this connection. 
When asked how a presidential candidate is nominated, 
60% of the high metro group said it is by a national conven- 
tion; only 29.7% of the^low metro group know thjs ^ 
procedure. At the local level, 95.3% of the high metro group 
know that local governments commonly collect garbage; but . 
only 76.1% of the low metro 17-year-olds rec(7gnize ihh 
social task as a local government function. 

.■\s is Sbservable in Exhibit 23, the relative positions of the 
(groups do not change greatly fpia the 17-year-old sample to 
the adult sample. The high metro group is high, the low 
mctr^f) \group is low^ and the remaining five groups con- 
(^re^ate closely about the median natiojial percentage of sue- . 
cess. ''^ o . 

.Mi high metro delta p-values- are abov^ the national 
percentages from 1.5% to 20.7^. All but two of these 
positive values are significant. All but two delta p-values of 
the urban fringe group are above the national per<^entages 
from 0.3% to 5.5%. The two Values below , the national ' 
percentages are -0.3% and -0.7%. Ab^ut half of the positive 
VfUues are significant. About half of the medium city delta p- 
values are abdve the national percentages from Q.4% to 4.7%' 
and about half are below from -0.3% to -2.4%. None is- 
significant. Over half of the delta p-valups pf the small place 
group are above the nationaK-percentages fronrv 0.1% to 
3,7%; the remainder drel^elow frdrn 0.1% to.4/6%. Three of 
those above an3 two Af those below are sigjiificant. All but 
two of * the ^treme rural group's values are below the 
national percentages from -0.5% to -13.3%; one is at the 
national percentage and one is 1.5% above. Five of the 
negative values are significant. About one third of the delta 
, p-values for the main big city group are above the national 
percentages £rorh 0.4% to 6.9%; three are significant. The 
remaining two thirds are below from -(1^5% to -13.7% with 
six significantly below. All of the lovrW^ro delta p-values 
are bielow the national percentages fromM^% to -28.2%. All 
but two are significantly below. 

As in the other age samples, a pattern emerges of a central 
core of five groups that behave homogeneously on these ex- 
ercises. With the exception of a half dozen results each for 
iht main big city, extreme rural and possibly the urban 
fringe groups, this core^roup deviates no more than ±5.0% 
from th^^national laercenbfii^es on all exercises.' The deviants 
are the high and low me^ro^4;pups\ With the exception of u 
haff dozen results between tn&m, these grouno Jeviate no 
le^s than ±5.0% fror^ the jiational percentages, on all exer- 
cises. ^ • 

The "central core groups resporisS similarly on many exer- 
cises: hut when the high and low nletro groups are included^ 
the adults respond alike on only one exercise. An unreleased 
question abdut the armed forces was answfi'ed correctly by 
96.3% of all adults. No group deviated more than ± 1 .5% from r 



the national percentage. The next nearest alike resppri^e is 
on the ballot exercise where respondents are askea if they 
can vote \or both Kirk and Jones for a stated open Senate 
seat. JMo group devfated from the national percentage of 
90.2% more than about ^.0% except for the low metro 
group. It is a significant -5.3% below the national percen- 
tage. 

On another of the general election ballot questions the 
difference between the high and low metro performances is 
illustrated. Among all adults, 70% interpreted the ballot ac- 
curately as Enabling them to vote for a councilman frdm 
each part^ More than 80% of the high metro respondents 
did so; but only 46% of the low metro group made^the cor- 
rect interpretation. ^ ^ 

On another exercise, about 60% of a^ adults indicated a 
knowledge of national cohventions. Nearly 80% of the high 
metro group know of the procedure, but only about 30% of 
the low mcifo group said they know presidential candidates 
are nominated in conventions. 

Neaiiy every exercise, it seems, is a potential example of 
the relative disadvantage of the low metro group in this as- 
sessment.^ The kinds of explanations one offers for the 
relatively poor performance of this group dependrpahially at 
least on his point 6f vitw and the se^ of concepts that view 
entails. One point of view mig))j depend heavily on a con- 

, cept of intelligence;*anoth6r mi^ht utilize broadef concepts 
of cognitive^ abilities; still another would use Concepts of 
culture and culture differences. The possible points of view 

tare- several and the concepts many.- ' 

One conclusion would necessarily Jiave to be accounted 
for w-hatever explanation is hypothesized. Whatever is 
measured by these exercises is measured with different 
. results in these polar groups. If some kin(J of ability attends 
^ ft suc^^ssful performance on tl|>se exercises — and that as- 
sumption seems a necessary one — then the high metro, 
group is more in possession of it than the low metro group,. 
What, exactly, that ability is is less-easily described than we 
might wish. That it is associated with sqhqoiwork seems a 
' likely inferendb.'That it is nurtured most effectively in an af- 
fluent environment seems also legitimately inferred. That it 
r is better to have than not to' have or that it is related to 
. anything else worth ha^sing are nubbier assumptions This 
assessment gannot help us with those. 

Rights and Duties. ^Ei^hibit 24 displays the median delta p-^ 
values for the s€Vcn STOC categories for the Knowledge of 
Institutional Rights and Duties in the three age Samples, *1 3, 
17 and adult. There are too few exercises to allow the inclu- 
sion of the age 9 sample in these exhibits. 

.Exhibit 24 fairly adeqaa^fely depicts the variance among 
these groups of 13-year-olds on the 10. exercises used to 
measure Knowledge of Institutional Rights and Duties. 
Mqre specifically, however, we have tried to note on which 
exercises respondents seerh to be quite similar across the 
* sample and on which exercises they tend to b^ quite dif- 
ferent. When we performed this analysis on the Structure 
questions, the high and low metro groups appeared dis- 
similar on most questions in all age samples. At the same 
time» the other five STOC groups appeared quite similar ori 
many questions. In this color scheme, all 13-year-olds are 
alike relative to one exercise. When asked if there should be 
laws against vandalism, 93.1% of all 13-year-olds indicated 
that there should be. No group deviated more than 1.4% 
from this national percentage. None of the deviations are 
significant. ^ , . * 
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On tlvo other questions^ one on due'-prpcess and the other 
on free assembly^ five of the STOC groups are 
^homogeneous. But 13-year-olds in the high metro group are 
significantly above these others by 7.5*JS and 10.5% and 
respondents from the small place group are significantly 
, below their colleagues by -3.5% and -4.3%. 

On three exercises related to religious ireedom, the 13- - 
ydar-olds arc not alike. For example, the 13-year-olds in the 
high metro and main big city groups are more likely to sup- 
' port for office a^person who does* not believe in God. Nearly 
60% of all 13-year-olds would allow such a person to hold of- 
.fice; 75% of the high metro group and about 63% of the main 
big city youths are of this mind. At the same time, only 55% 
of the 13-year-olds from small places anff^46% of the low 
metro youths • support this belief. On the other two un- 
released religious freedom questions, the variations arc dif- 
ferent, though the high metro group register^ consistently 
higher percentages than any other group. The extreme 
rural, low nietro and. small place groups are significantly 
below the national percentajges on one or the other of thc€x- 
ercises. ' V* 

Four of the STOC groupV, medium city,c?urban fringe, 
main big city and low metro^ show no significant variance 
from the national percentage of 88.8% on thtf question of 
racial di^rimination in employment. The other three 
groups,^owever, vary significantly ddthis question — small 
place residents by -3.2%, extreme rural by -4.1% and high 
metro by 7*4%.' * . ' 

Fewer than half of ^11 13-year-olds believe m or recognize 
the freedom of the press in politically sensitive areas. Both « 
extreme rm al and low metro groups;arc even below this 
level at 36% and 39.3%. The high metro group is ^ 
significantly above ^the national percentage and all other 
G;roups at 61.5%. Freedom of the press gets short shrift 
amone; 1 3-y ear-olds, as do scvA^al other basic democratijc 
rights and freedoms. \ 

By 17 years-of age an overall gain of 4% is notfed relative to 
the 13-year-61tl median national percentage. To the extent 
that the two sets of exercises are comparable,. the gain may 
be interpreted as an increase in the understanding of our in- 
stitutional rights and duties or possibly an increase in com- 
mitment to them. But also, as can most easily be seen in Ex- 
hibit 24i there are some shifts in the relative position of some 
groups. The most notable of these> according to an analysis 
of exercises, is the exchange of extreme rural and low metro 
groups in the position of exhibiting the lowest median delta 
p score. The main big city and high metro groups also shift 
downward. However, because the main lig city*s median 
delta p-value stays clustered with those closest to nationaf 
percentages, its change In status does not make so much dif« 
ference on most exercises. Also, since the hjgli ipetro group^ 
regains its highest position relative to a national perccnts^ge, 
itf distribution is not much different. ^ 

Within the 17-year-old sample, the high and low metrd 
and extreme rural groups are most deviant. We will 
emphasise these groups in the discussion and refer inciden- 
tally to the othec groups, which tend to be fairly 
homogeneous. » 

In an earlier section we referred to a set of two questions 
as seeming to have a law ai|;id order orientation. It included 
the o(^ten-cited exercise on vandal laws and an unreleased 
question on crime reporting. In this age sample, extreme 
rural youths record their highest delta p-values on these ex- 
ercises. The small place group almost equals the extreme 
rural youths* special concern for these matters. At the op- 
posite end, the high metro group places these two questions 
in low priority. It achieves significant negative delta p-values 



on both. That in itself is notable, for the high metro group 
has very few delta p-values below the national percentages. 

More characteristically^^^the high metro group has positive 
deha p-values ranging from 12% tt) nearlyM6% on a set of 
five exercises that seem to measure preferences for open 
protest, freedom of the preriss and freedom of religion. We 
suggested earlier that these^questions have a liberal orienta- 
tion. The 17-year-olds ^s a total group are not overly fond 
(or apprised) of these ideas, but rural and small place 
youths find them especially distasteful (or unfamiliar). 
Aboiit 45% of the small place group and 35%'of the rural . 
group» for example, would permit a police station to be 
picketed. On this exercise the low metro achieves its highest 
deha p-value; 57.8% of that group would allow such a 
protest, Low metro youths do not tenJl to extend this right to 
include picketing of rocj^^sttvals, however; only 46.2% 
would allow that action. The high metro group docs not dif- 
f<&rentiate betweeh these objects; nearly two thirds would al- 
^w both a polide station and a rock festival to be picketed. 

The hardest question Sr 17-year-olds tested their 
knowledge- of the Supremp Courtis decision <m religion in 
schools. Only half marked it is relatin^to the principle of 
» church and state separation. The urb^n fringe group did 
best of all groups on that question; nearfy 60% could answer, 
it correctly That amounts to a 20% advantage over the low 
metrp group, which did the poorest of all g^-oups on the 
question. 

The easiest question for 17-year-olds relates to race as a 
factor in employment. It is also one offtthe few questions on 
which the,^l 7-year-olds are much alike. Except for the main 
big city group, which produced a deha p-valuc/of -4.3%, all 
groups are within 2% of the national percentage of 93.8%. 

At the adult level, the mojrt Interesting comparisons are 
again among the high and low^nietro an(| extreme rural 
groups. The interest derives from their extreme variance on 
a number of exercises. Those variances are reflected in the 
median values reported in jExhibit 24.*The four central 
groups, urban fringe* main big city, small place and 
medium city, vary from the national percentages on in- 
dividual exercises no xftorc than ±8.2%. In contrast, the 
extreme rural adults have a dispersion^ of 25.5% running 
from 3,8% to -21.7%; the low mctro^group goes from 6.4% to. 
-20.8% for a dispersion of 27.2%; and the high metro group 
has a low delta p-value of 0.0% and a high value of 25.2%. 

Nine out of 20 exercises in this color iScheme have been 
released. As a moans of illustrating th^ wide variances 
among these thfcee groups, 6 of these 9 exercises are listed 
below in Table 10 along with their national percentages of 
success and the delta p-values for the high and low metro 
and extrenfe rural groups*. An * beside a delta p-value in- 
dicates that the difference from the national percentage is 
large enough to be statistically significant. 

The point of the display in Table 10 is to illustrate that 
ijie aduhs in these three STOG groups are indeed different ^ 
in th^ way they, perform on the Rights a'nd Duties exerci&es. 
The groups also appear to vary from each other op the 
remaining 14 exercises in the color scheme. The extreme 
rural and low metro groups comf closest together on the 
question concerning the right to assemble. Only 4 percen- 
tage points separate the resuhs. On three exercises they are 
separated by about 10 percentage points, land on one they 
are 28. 1 percentage points apart. TTfe three groups are most 
nearly alike in their responses on the question about racial 
discrimination. On that question, the' high and low metro 
groups are alike, i.e., their deviations from the national 
percentage are not significant. 
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TABLE 10, Comparison )f High and Low Metro and Extreme Rural Adults 
on Selected Rights and Duties Exercises — 
Social Studies 1971-72, Delta P 



Short Text of Exercise 



Allow picketing of police station 

Supreme Court decision on religion in schools 

Allow atheist^to hold public office 

Allow picketing of rock festival 

Assemble inpaVk for protest 

Race a factor in employment 



National 
Percentage 


Extreme 
Rural 
Delta P 


High 
Metro 
Delta P 


Low 
Metro 
Delta P 




52.2 


-21.7* 


16.5* 


6.4* 




52.3 


-8.8* 


17,4* 


* -18.3* 




55.6 




25.2» 


-3.8 




63.0 




15.1* 


.20.8V ^ 




77.5 


-13.8* 


10.3* 






92.3 


-.6.3* 


0.2 


^ 2.3 . 





The high metro and extreme rural groups are most dif- 
ferent in their responses to the queslion»on picketing police. 
\ Over two thirds of the high metro adults would allow that; 
only one third of the extreme rural, adults would. The high 
and low metro groups are most different in their responses 
^ to picketing rock festivals. Where 78.1% of the high metro 
adults would support such an activip/, only 42.2% of the low 

metro adults would. ^ - . 

An iipplication of these differences is that more than a 
single factor affected the responses of the three groups. 
Picketing as an activity seems relatively equally accepl'able 
to high metro adults vci very different contexts. It seims 
fairly unacceptable to extreme rural adulli in either circum- 
^ stance, but more so in one than in the* other. It seems more 
acceptable 19 low metro adults when directed against police, 
than when directed^against rock festivals. A social comment 
with several facets seems buried in those .results. , *^ 

Summary 

We attempted' as an overall strategy to divide a selected 
set of 48 exercises from the national assessment of social 
studies of 1971-72 into two parts. One part is comprised of^ 
items that we judged to deal mainly with various structural 
elements of institutigns. ^The other part is comprised of 
items that we judged to deal primarily with rights and 
duties within these institutions. Though we persist in using 
. the broader term ojf "institutions/' most of the exercises, as ' 
it turns out, relate mpst directly to^the institution of govern- 
ment- Some of the same exercises "w«re u-icd as the basis of 
^ the NAEP publication, Political 'Knowledge and Attitudes W7U 
1972.^ • ■ ^ * 

In retrospect, the division of exercises into Structure and 
Rights and Duties color schemes appears to have had only a 
visual warrant since the two sets of questions produced very 
similar statistical discriminations among the various 
^ categories of persons assessed. That makes one believe that 
similar factors operated in the two parts; the 'two color 
Schemes, in effect, appear to have assessed the same thing, 
whatever that is. We could continue to insist obdurately — 
which we do — that the subject matter division is valid, 
visually if not otncrwise,, and that the samples pf persons As- 
sessed ai^e in roughly equal possession of the two contents. 

The parallel results along with either of these explana- 
tions can be thought of as partial justification for another 
decision ma^ about the content of the tyvo color schemes. 
Both setr> of questions were approached as if they measured 



knowledge. That decision will not bear up under a visual in- * 
spectioh of exercises, however, because some questions, es- 
pecially in the Rights and Duties color scfi^qje, contain the 
value term **should" as the interrogative. Our a priori 
justification for largely ignoring this elementary distinction 
in these questions is argued elsewhere. Mainly, that 
Justification suggests that the involved questions refer to 
fully institutionali^d rights-in-ihis society and do not, in a 
real sense, pose issues. These questions, the argument con- 
tinues, may just as validly be assessing these persons' 
knowledge ''of these rights as their preferences for fhem. 
While Folilical Knowledge and Attitudes ^treats some of these 
questions as outright value questions, as do we guardedly in 
several of' our discussions, the parallel results of the two- • 
scheme analysis used in this report seem to adfi some 
measure of justification^ to ignoring the distinction for this 
assessment. 

An assumption was invoked On occasion in this analysis 
that reaultiacross age samples imply growth or change. The 
use of line graphs to depict the median perforniances of the 
categories of persons at different age levtib' appears to en- 
dorse this assumption. The truth of the matter is that we do, 
not know for certain whether the performance of any ageiS^ 
level in this assessment can be projected either forwip'd or^ 
backward on the basis- of the performance of any other age 
level, even though they may shdre severar questions. 
Therefore, all statements that suggest br state outright that 
the 17-year'^oIds learned something of this content between 
the ages of 13 and 17 or that the adults have not learned 
much of this information since their school days must be 
read more cautiously thanlhey were written. We found it 
useful to hypothesize these. projections, but more and dif- 
ferent information than is available fro9i t4iis assessment 
would be needed to say such tjhings reliably. ^ ^ 

Even with^all the analyses, categories, cfclor schemes and 
age samples, rt is still most difficult to account for any of the 
results in the assessment. The reasons^ for this interpretive 
impasse are so well\nown by thoughtful school people that 
it seems gratuitous to smention them. Interpretive impasse 
notwithstanding, occasionally we did cast what could pass 
for an explanatory Hypothesis; albeit, at some later date we « 
may ne^d to seek forgiveness for some of them. What we 
have here is a mass of descriptive data, confounded in all the 
ways that such data are always confcJUndedi in spit^ of 
NAEP*s meticulosity, and in which exists not one explana* 
tion. 
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' A Jew content generalizations appear to emerge with 
some consistency in the themes reported in this analysis. 
•One that comes to mind is the recurrent emphasis on law 
*and order and the protection of property. Whatever the 
reasons for ."^thc high acceptance.^ of tficse values, their i 
emphasis in this assessment by^^many groups suggests 
something, other than a -general ^breakdown in the concern 
for security in this society. Another emphasis that seems * 

- clear is the hjghi agreement by most groups that race should 
not be a factor in employmiint. That would seem to repre- 
sent a major change in the social preferences in this country 
over the past decade or so. 

We, of course, can st^te sevccal fadts with some firmness - 
about the catcgori« of persons assessed. We can say that ^ N 
persons of all Sges i.e'., ages''9, 13, 17 and adult whos^ 

. parents went to college, persons of all ages who live in 
rel^ttive affluence, persons who live in the Northeast part of 
our country, males at age 17 and as adults, and whites at all 
ages are the most able of all categories of persons sampled to ^ 
respond according to the criteria of Ihis assessment. 

We. can also say, given the content,' dcfmitions and 
criteria of this assbssment,'that persons whose parents never 
went beyond the eighth grade, persons who live in thejinner 
city, persons who live in the Southeast part of thig country, * 
females and blacks do not score as well on t^ese exorcises as * 
do persons in most other categories. The data tell these 
facts» but they don't answer the important questions. 

^The impoftanTquestions fall in't\vo categories. In the first 
of these, the' questions stab angrily ^nto the inner 
mechanisms and justifications of NAEP. Other researchers ^ 
in thisjsocial studies project have raised some of these im- 



portant questions. Others have been raised in different 
forums. This report has perhaps raised fewer than it should. 
One stands out as signally important at the moment. Is it 
possible to construct an assessment that examines validly,", 
reliably and fairly a plural population? There seems ob-* 
viously a sense in which pluralism should dictate the stan- 
dards that are Applied in such a mass accounting of our 
schools and ourselves. Our differences and diversity surface 
in this social studies assessment as they do in ^11 such assess- 
ments. But we seem easily tempted to judge the reasdnable 
responses of others who enjoy different advjBitages than our 
own as less reasonabte than our own. 

In another category, the important questions seem mostly 
to test our social understandings and motivations. Why do 
these groups of persons brfiave differently on this assess- 
ment? The assessment's yield of dcsmptive data won't suf- 
fice to. account for the differences. Do we want — m^re im- 
portantly, 'should we want — such groups to perforin more 
, evenly? If we do, how can schools lielp to make it happen? 

The two categories of questions tt|rn rapidly i^to each 
other. While we assume that some of the diversity , exhibited 
in the assessment is a function of the assessment and some a 
function, of schooling, beyond these, much of^ it is un- 
doubtedly a product ol^this society. If for some reasons we 
desire a population that performs more evenly ii^ such assess- 
ments, altA-ations that go beyond the unfiiasing of a test 
and that go beyond th<i equalizing of schooling are likely to 
be necessary. Given the apparent value conflicts in this 
society and the different frami^s of reference that are dis- 
played by tlys population, such a desire woqld seem empty, 
even if it were justifiable. 



NOTES 



1. Sec Appendix D> Reference 4, pp. 9, 40-41. 

2. See Appendix D, Reference ^12. 
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CHAPTER 9 



A RESPONSE FROM NATIONAL ASSESSMENT 



J. Stanley Ahmann . 



The 36 social studies educators who participated in tl>c 
NCSS review of the first assessments in citizenship and 
social studies are to be commended for their thoughtful 
analysis and suggestions for * improvement. The report 
provides valuable input for decisions to be made in the year3 
ahead. In commissioning the study, NAEP's purpose was 
twofold: to bring the assessments to the attention of (he 
social studies education community via a thorough ap- 
praisal by kfey personnel from its own ranks, and to lay the 
foundation for further cooperation bctwicen our two 
organizations. We feel tKat <hese goals hav^been achieved 
although there are still hurdles to be surmounted, before all 
will be satisfied with the end results. 

The National Assessment of ^Educational Progress 
(NAEP) has a primary ^purpose to monitor change over time 
through the collection of education stati^ics at the national 
level. This longitudinal aspect means that the 1969 
citizenship.and tjiic 1971 social studies assessments provide 
only baseline dAia^ although some hypotheses may be 
.drawn as several of the authors havesu^gest^d. Because the 
objectives on which these ai^sessments Verc based resulted 
from a broad consensus, they reflect what the majority, at 
the time, considered desirable goals aod.bchavior jn these 
two areas Qbr the majority of the population. This fact 
should not disturb those who do not hold these same goals 
nor should it be construed as saying anything about the 
merits of alterni^te philosophies. Measurement of all phases 
of a pluralistic society is a financially impossible feat^ and 
, National Assessment does not attempt to do this. Because of 
the dynamic nature of the NAEP models however, there will 
, be ample opportunity for changes in the nature of these ob- 
jectives whenever the majority so determines. * " 

In the dozen years since the concept of a national 'assess- 
ment was first debated, ideas, coqriments, criticisms and 
compliments have come- to the project. To date almost a 
thousand scholars, educators and laymen, plus almost half a*^ 
million" youth and young adults l^ave participated in the 
development of objectives, the review of exercises and the ac- 
tual surveys in the 10 learning areas. As the second cycle of 
assessments, begins, a number of major chan|;cs are in 
progress. " \ 

During the past two years the federal-govermcnt has cut 
funds for HEW projects, incIiiding NAEP. WitK the current 



economic situation the NAEP budget is not likely to return 
to its peak period in fiscal 1973 for sevaral years. As a result, 
certain modifications of the orlginaf model haye been made. 
For exartipfe: ^ 

1. The 10 learhing areas have been consolidated into 5,-^ 
science, mathematics, reading, humanities (art, music 
and literature), and citizenship/social studies. Writing 
and career and occupational development (COD) will be 
assessed as special probes jn the sense, that the asses^- 
ments will be limited to subsets of the field — i.e., writing 
will be limited to writing mechanics and the special 
probe in COD will be administered Qnly to l5^,yealr-olds. 

2. Beginning.in fiscal 1975 the adult si^mple has been drop'- 
ped, and starting in fiscal 1976 (when die citizen$h>p/ 
sQcial 'studies reassessment will occur) the out-of-school 
17-year-olds will also be eliminated from the;^,3ample. 

3. The total number of cjcercise packages must h« reduced 
although both individual and group exercises will con- , 
tinue to be administered. 

4. The number qt selective re'ports for each learning area 
will be limited to three. , * * 

5. Staff expansion has been curtailed and the practice of 
subcontf'^cting in all areas other than data collection and 
scoring has been eliminated. ■ o ' 

Hov/ will the reasses^m&it of citi2;ens hip/social studies 
differ from the original assessments? The citi:2enship and 
social studies objectives were reviewed in 1969 and 1972, 
respectively, preparatory to the development* of new exer- 
cises for thi second-cycle assessments, which were to Jiave 
taken p^ce In 1974 for citizenship and 1977 for social 
studies; Readers will note tbat^a number of the concerns ex- 
pressed in the NCSS- sflidy were anticipated by those 
engaged in this review. The following excerpts from the 
revised objectives koklets summarize the nature of the 
changes. A complete list of both^he original and revised ob- 
jectives and subobjectives for each learning area can be 
found in Appendixes A and B. 

Citizenship 

The main changes irt the citizenship objectives prepared 
for the 1974-75j^asses'smcnt may be summarized as follows: 
1 . For sbme ^ubobjectives, reviewers identified im- 
portant omissioj[is at the 9- and l3-.year age levcjs. 
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Subobjectives and appropriate illustrative 
behaviors have b(!en added for these age levels in 
sevclral instajices. • , 

2. The- substantive, nature of civic problems was left 
iropKcifni the original t)bjcctivesfor^hemost^art, 
except that international problems were spelled 
out in detail. The substantive description of major 
civic" pKo)>lerhs which a citizen needs to understand 
^in order to act effectively has been ejfpanded. In 
addition to extending problems- in'social conflict to 

^ include local and national problems, two other 
major problem areas affected by civic policy, 
namely, economic needs (poverty, employment, 
ctm) and'^environ^nental problems (pollution, etc.) 
have bteen specified. * 

3. In response to criticism that the objective on 
knowledge of governmcrit was too, limited to text; 
book ideals, illustrative behaviors wefe ^dded con- 
cerning knowledge of informal influences on' 
government, sources of ' actual ^powcr and 
bureaucracy. Effectiveness of citizen particifiation 
rather, than effort alone was also emphasizeor 

4. Objectives dealing with personal development and 
voluntary personal relations were in many in- 
stances concluded to be too remotely related to 
citizenship to retain. These aspects of citizenship, 

while not eliminated, received less emphasis in the 

11-. . ' ft 

revised objectives. > 

5f The behaviors' listed under a jiumber of the 
original objectives seemed to emphasize middle- 
class values which might ndt be accepted as goals 
f>y other social strata of the nation. In several in- 
stances such behaviors were deleted cr changed to 
example^ with more universal appeal. For exam** 
pie, "controlling eniotions in the face of criticism," 
, was changed to "express emotions in nondestruc-^ 
tive ways."^ 

The objectives are intended to be a-^ working guide 
for the dii^Oicult task oi^^ssessment, not a despription of 
ho^ an ideal citizen^ should spend liis day. No one 
person could be expected^ to exhibit all of the speicifib 
behaviors included. But since the assessnient is in- 
tended to describe the achievements of a wide popula- 
tion qf citizens, not individual persons, this present^ 
no problerh. For example, it might be reported that, 
"10 percent of 17-year-old boys in ^e country have 
served as leaders, of a groUp engaged in civic ac- 
tivities.** Such a result would not be intended to- dis- 
parage in any^ way tho^e 17-yeaf-olds who felTtJiey^ 
were not qualified to act in such a role or.taimp^ that , 
"the more time a person speji^s.at civic pursuits, the 
better." ■ 

National Assessment objectives are not permanent 
standards of achievement; rather, they .are intended 
to reflect the evolution of goals in e^tfCatioli in 
response to the changing needs of^tl^ nation.' 

Social Studied 



During the winter of 1972-73, thc^s'ocial studies objeotives 
deh'neated in this booklet jwere fofmblated'by social studies 
an^ social science specH^lfsts from various universities and 
secondary ^and elemejitary schools^ and by lay people in 
various occupations from differents parts of the country. 

One of the challenges in the proceSs of developing objec- 
tives for. the next assessment of social studies has been the 
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nefces?ity for .§ubject-matter specialists- and lay people to 
. develop a framework of major objectives that would allow . * 

^ for tl>e^ development of- niore specific subobjectives and 
« related age-specific illustrations. It vyas decided that an ap- 

- propriatu fra^mcwork-for the measurcmem df ach ie vement in / 
'the social i^tudies would inqlude the acquisition . of a * 
knov^ledgl: base for understanding human bein^i and their 
relationships with their environments, an^ understanding of 
values as tl^ey relate t.o individuals^ and groupV, the use of in- ' 
tellectuai and hliman relation, skills, the development of a 
positive self-concept .and" a sense of and commitment to -s 
rational social parjticipatidn. ^ " • ' . ^ _ 

The- knowledge to be acquired-'ih s<H[ial studies 

^ «dr^ws.onuhe content of the majof social science dis- . 
ciplines but is organized in terms jnterdisciplinary 
* concepts and ideas td allow a broader and more in- 
tegrated'^pproach.to.the study of social phenomena. 
The skills to bt i^q\i}red are those of intell^ctU2\l' in- 
quiry and huma^L relations tjjat enable the learner to v 
/-ask questions about/social prpTalems apd to par- ^ . * 
\' ticipate rationally apd responsibly in societyt.Ai^ lin- 
• .derstaridtrig of the ^^alues'6f individuals and*' groups . 
» are includedun recognition of the nepd'to''explore and ^ 
, '.iclarify v^liie' orientation^ that underlie our institti- * 

' . tions ^nd those of , other' .societies. ItMs also ap- 

; . propriate'tbTecognijje that a person's self-percep^iori 
* .js closely^ relateo (a- ffie individual's rdle in society^."^ 

. Since ^Ocial'Studieis. deals with the study of t^ie in- ' 

. • dividu^rs role if>''qoci6ty, thp development of'^a ^. * - 
, ^„ positive self-concept -becomes *aV in^portant objective 
Vn-thi§^area. I'He^'iolporbncc^ofvthc p^rtjcLpatiortof * • 

. young people. ii>- the sokitio^ri of societal problems 
, prompted th^.inclusion .of a fipal objective -that ad-' ^ • 
dresses itself Jo XYig cpiprghment >to t)ie right ;of self- * * . 
- deteVminatio^ io^ al! hum<iri beings.and A' williihgrfes^, . 

to take rational a?ti^';in,support,pftme9ns for secur* ^ ; 
^ ing sffid preserving "hyman tights.^ J* , * ^ 
^Whfcn the decision was madcvfa combine citizenship And - / 
Social Studies in one ass^ssmf^i uiider the revised time 
schedule, the dtizenship^Vedevfclci^ment .process had b^^n * -..i 
virtually completed. The social Studies redeyejop^ent had^* 
only progressed as far as the st^e* of objectives; review. \ / 
Therefore, the 1975-76 assessment ;>vill'be predominantly a 
citizenship one plu^ a reasscssipenj l^f iVh'e original (xn^ ' 
released social studies exercises. New .citizenship exercises 
will reflect the recommendations of the Hujakins coVnpiittee 
for greater emphasis on upper cognitive skills and more'ex- ^* 
ercises at the affective level. FurthernT|Ofe, future,, social 
studies exercises are expected to take in'to account' the 
results of a survey of social studfes educators/concerfiii^g tjie 
amount of weight each subobj^ctive and objective snoiild 
carry. Whife a subobjectiVte may still have only a lin^ited 
number of jtxercises, by choice, the overall objective wijj 
have ample Coverage. ^ , 

Specifically, the 1975-76 citizenship/social studies assess- 
ment will consist of 49 hew citizenship exercises, 55 original^' 
citizenship and 22 original social studies exercises. When 
overlapped administrations across the three in-school age 
leVels are counted, the result will be approximately 360 exer- 
cises. One fourth of the new exercises and two thirds of the 
old ones will be released aftej* the assessment. The 
remainder will be the Jaasis of. the comparison data for the 
third assessment. 

Since the chief functipn of NAEP is to me^Sijre change it 
is imperative that exercises used for comparsion purposes be 
identical to their original form. Although a few unreleased 
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exercises have been dropped from the second fcycle, the bjalk 
of them will reappear. While this action may discourage 
those who_ would prefer tp ^ee somf^hanges in the )vording 
of certain exercises, it shoulcl be noted lhat no exel-cise'is 
used that has xiQt passed an ex>«n^ve screening process. The 
NAEP exercise-review process J^s been increased fourfold 
Tom that used for the first assessn^t: It is.anticipated that 
the exercises »that su^h^ive this screening Will meet with 
greaPer approval, * 

What does jhe future hold for JSIAEP? Citizenship arid- 
social ^tudies will remajn a single assessment, but attention^ 
will neeffi'to be given to a blepding of the two. Furthermore, 
among the related ongoing research efforts by staff is an ex« 
a^ninijtlon of other background data that might be coltected 
about <t he individuals participating in the survey, ^vhich 
^(Aul<l aid those who must make decisions about the findings. 
^Ope^such study has already resi, !ted in the monograph As- 
socfadons Between Educational Outcomes and [Background ^ 
Variables: A Review of Selected Literature , and staff are engaged 
in (5leveloping.a plan that will incorporate some new features 
— but not for the 1*975-76 assessm'c-^^. 



Documentation has been completed regarding the in- 
dividual responses for all released social studies exercises 
from the first assessment angUmay J)e obtained by indepen- 
dent researchers from ffie ED STAT 11 division of the 
National Cpnter for Education Statistics in Washington^ 
p.C Other computer tapes already on file there include 
NAEP reading ihe literature responses. .Citizenship files 
frbtn the first assessment are not available, but citizenship- 
/^ocial'Studies 'files from the second assessment will be ad- 
ded as soon as possible after the completion of the data 
analysis. ^ , ^ 

Hopefully, conditions will one day alkjw the re§umption 
of data collection for adults and out-of-school 17-ycar-olds. 
Meanwhile, we shall continue to call upon professional as- 
sociations to help identify strengths and weaknesses in 
National Assessment. This NCSS report clearly 
damon^fral^s the helpfuj^informa|ionthat can result from a 
thorough evaluation of assessment data and the objectives 
and exercises behind them. The entire^, education com- 
munity should benefit from the careful wpjrk of these 
members pf the National Council for the Social Studies. 



NOTES 

1. See Appendix D, Reference 2, pp., 4-5. 

2. See Appendix D, Reference 15, pp. .v-vi. 
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APPENDIX A 
CITIZENSHIP OBJECTIVES 
1 969-70 Assessment 



^elfarc and Dignity of 



I. fShoV Concern for Ac 
^ > Others 

A. Treat all individuals With respect. 

B, * Clonsider the consequences for others of their 

own actuuis. 
^ C. Giiard safety and health of others. 

D. Help other individual^ voluntarily. 

E. Arc loyal to country,! to friends, and to other 
groups whose values they share. 

F. Understand and oppose* unequal opportunity in 
the areas of education, housing, employment, 
and recreation, 

G. Seek to improve the Welfare of groups of people 
less fortunate than t|hey. ^ '.y"^ 

II. .Support Rights a^d Frt^edoms M All Individuals 

A. Understand the vajue of constitutional rights 
and freedoms. 

B. Recognize instances of the proper w^crcisc or 
denial of constitutional rights and liberties, in- 
cluding due process of law. / 

C. Defend rights and liberties of all kinds of people 
uniformly. I 

III, Help Maintain Law iind Order 

A. Understand the need for law and order. 

B. Are conscious of right and wrong behavior, 

C. Comply \vith public law and' school rules. 

D. Help authorities jin specific cases. 
^ E. Protesjr unjust rijles openly. 

Inforp themselves about the law. / 

IV. Know tlie Main Stifticture and Functions of Our 
Governments | ^ \ 

A. Recognize the fjurposes of government. 

B. Recognize the fe^in functions and relations of 
j governmental bodies. 

C. Recognize the importance of political opposition 
and diverse interest groups. 

D. Recognize that democracy depends on the 
. alertness and involvement of its citizens, and 

know how citiiens can affect government 

E. ' Recognize the structure and operation of 

political parties. 

F. Know structure of school and student govern- 
ment. * I 

' ' ' . / 



V. • Seek Community Improvement Thro^g^l Active, 
Democratic Participation 

A. Believe that each person's civic behavior is im- 
portant, and convey this belief to others. ^ 
Recognize important civic problems and favor 
trying to solve them. 

Actively work tor community improvement 
Participate, in iQcal, state and national 
governmental processes. ■ , 

Apply democratic procedure? on a practicat 
level when forking in a group. 
Display* fairnes^ and good sportsmanship 
toward others. ^ 



B. 

C. 
D. 



F. 



VI. Understand Problemi of International Relationi 

A. Are aware of the problems of international con- 
^flict and dangers to naticyial-security. 

B. Seek world pe$ce and freedom, for all peoples. 
• ■ ' I ■ ■ ^'^ • 

VII. Support Rationality in Communication , Thought 
and Action- on Social Problemi 

A. Try to inform jhemselves on locially important 
matters *and to understand \altcrnative view- 
points. I * \ ^ 
Evaluate cdmmunications critically and form 
their own opinion^ independently. ^ 
Weiglv alternatives and consequences carefully, 
then make decisions and carry thtm out without 
undue delay. . \ 

D. See relations among social problems and have 
. good ideas for solutions. ^ \ > ^ 

E. Support free communication and communicate 
honestly with others. \ 

F. Understand the role of education in developing 
good ditizens. 

VIII. Take Responsibility for Own Personal Develop- 
' ment 9nd Obligations 

A. Further their own self-improvement and educa- 
tion. ^ 

B. Plan aheM^for major life changes. 

C. • Are conscientious, dependable, self-disciplined, 

and value excellence and initiative. 

D. Economically support self and dependents. 
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IXZ-^Hclp and Respect Their Own Families (Ages 9, 
13, 17) • 

A. Respect the reasonable alithority of their 
parents, or guardians, and help wj|h home 
duties and problems. 

B. Help younger brothers and sisters^^to develop 
into good citizens. 

C. Discuss social m^tter^^ith their families and 
. respect the views of all family members. 



0 



X. Nurture the Development of Tbeir Children As 
Future Citizens (Adults) , 

... - (•• — - — ?.-,...» 

A. , Provide for the basic needs and health of theif 

children. * * 

B. Encourage cooperative, ethical relations to 
authority and to other individuals. 

C. Develop in their children a broadening 
awareness, independence, and rationality. 



SOCIAL STUDIES OBJECTIVES 



1971-72 ASSESSMENT 



L Have Curiosity About Human Affairs 

D 

A. Identify ^nd defme problems and issues^ 

B. Formulate generalizations and ^hypotheses 
,^ capable of being tested. 

C. Obtain information from a variety of sources. 

XX Distinguish facts from opinion, relevant from ir- 
relevant information, and ^eliable from un- 
reliable sources, 

E. Detect .logical errors, unstated assumptions, and 
unwarranted assertions; question unsupported 
generalizations; are aware of the complex 

' nature of social causation and understand that 
sequbnce or relationship does not necessarily 
imply causation. . , 

F. Use data and evaluative criteria to make deci- 
sions. • 

II, Use Anftlytic-Scientihc Procedures Effectively 

^. Raise* questions and answers, 

B. Are open to new information and ideasr 

C. Try to understand Why other people think and 
^act as they do. ^ 

III. Are Sensitive to Creative-Intuitive Methods of 
Explaining the Human Condition 



A. 
B. 



C. 



D. 



Read history, philosophy, and fiction. ^ 
Obtain insight into human affairs from history 
and*' philosophy, and from fictibn and other^ 
forms of art. 

Recognize the role of creative-intuitive methods - 
in scientific inquiry. 

Distinguish personalized explanations of human 
affairs from scientific-objective explanations. 



IV. Have Knowledge Relevant to the* Major Ideas 
and Concerns of Social Scientists 

A. Understand some of the distinctive modes of in- 
quiry (questions and approaches) of social 
scientists. 

B, Understand sbme 9f the; major relationships in- 
volving culture, the group, and the self. . 

C. »^Understand somoh&f the major charactefistics of 

economic systems, expecially the American 
economic system. ' 

D, Undft-stand some of the major characteristics of 
the geographic (spatial) distributions of nian 
and his activities, and of man's interaction with 
the physical environjnent. 

E. Understand some of the major historical 
developments. 

F, Understand some of the characteristics of the 
major systems of government, particularly the 
political system of the- United States. 



V. Have a Reasoned Commitment to the Values 
That Sustain a Free Society 

A. BHieve in the fundamental worth of the in- 
dividual and can justify their belief. 

B. Believe in the freedoms of the First Amendment 
and can justify their belief. 

C. Believe in the rule of law and can justify their 
belief. 

D Believe in open opportunity for advancement 

and can justify their belief. 
E. Are willing to act for the general interest, 
p. Are^'willing to participate in decision making 

relevant to their lives ^ 
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Afi't'ENDIX B 



REVISED CITIZENSHIP OBJECTIVES 



I. Show Concern for the Well-Being and Dignity of 
Others , ° ^ 

A. Treat others' with respect. 
•B. Consider the consequences for others of their 
own actions. 

C. Guard ^safety and health of others. 

D. Offer help to others in need. 

E. Support equal opportunity in education, hous- 
ing, employment, and recreation. 

F. Are loyal to country, to friends, and to other 

* groups whose values they share. 

G. Are ethical and dependable in work, school, arid 
social situations, . , 

II. Support Just Law and the Rights of^ All In- 
dividuals 

A. Understahd the need for law. 

Recognize specific constitutional rights and 
liberties. 

C. Defend rights and liberties of all kinds of people. 

D. Encourage ethical and lawful behavior in others, 
fi. Comply with public lawk ^ 

f . Oppose unjust rules, laws, and authority by 
lawful means. 

IIL Know th^ Main Structure and Functions of Their 
Governments • , 

A. Recognize basic governmental purposes. 

B. Understand the organization of federal and state 

• governments. • 

C. 'Know the political structure of. their com- 
* munity. 

D. Recognize the relationships of different levels of 
government. 

E'. Recognize the importance of political opposition 
and interest groups. 

F. .Recognize that democracy depends on the 
alertness and involvement of its citizens, and 
know how citizens can affect government. 

G. Know structure of school and student govern- 
ment. 



. IV. Participate in Democratic Civic' Improvement 

A. Believe that each person's civic bvehavior is im- 
portant, arid convey this belief to others. » 

B. Favor organized civic action where it is needed. 

C. Actively work for civic improvement. 

D. Participate in local, state, and national 
governmental processes. 

E. '^Appiy democratic procedures effectively In 

small groups. 

-V. Uhderstand* Important World, National, and 
Local Civic Problems 

A. Understand social conflict among individuals, 
groups, and nations and the difficulties in 
achieving peace and social harmony. 

B. Recognize how different civ|c policies may affect 
people's efforts to meet their economic needs. 

C. Recognize major environmental problems and 
are aware of alternative civic solutions. 

D. ^ee relations amopg civic problem^ and par- 
ticular events. ^ • . * * 

^E. * Generate good ideas about causes and solutions 
. for civic problems. ' ^ * 

VI. ^I^roach Civic Decisions Rationally 

A. Seek relevant information and alternative view- 
points' on civically important decisions. 

B. Evaluate civic communication.^ and actions 
^ carejully as a basis for^ forming and changing 

{heir own views. 

C. Plan and organize civic tasks. effectively. 

D. Support open, honest communication and un- 
rx4rsal education. 

VII. ' Help and Respect Their Owo Families 

A. Cooperate in home responsibilities and help 
provide for other family members. - 

B. Instill civic values and skills in other family 
members. 
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REVISED SOCIAL STUDIES OBJECTIVES 



S-Y^AR-OLDS 

Develops a kiio|t^ledge btie for understanding 
the relationships between human beings and 
their social and physical environment. 

A. . Acquires kjowlcdge about social organization. 

B. Acquires knowledge about the relationships 
between human beings and their social environ- 
ments and understands some of the conse- 
quences of these relationships. 

C. Acquires knowledge, about the relationships 
between human beings and their physical en- 
vironments and (Understands some of the conse- 
quences of these relationships. 

D. Acquires knowledge about decision-ma*'*ng 
processes. • ^ 

E. Acquires knowledge about conflict and the im- 
pact it has on individual' and group 
relationships. 

Develops an undenitanding of the origins and in- 
terrelationships of beliefs, vajues and behaviof 
patterns. — * : ' " 

A. . £)q)resses awareness of some of the beliefis and 
^ v'arues expressed by people and recognises that 

the conditions, times and places in which pepple 
live influence their beliefs, values and behaviors. 

B. Unders ^nds ways beliefs and values are trans- 
mitted in various cultures/ 

C. Understands some of the influences XA differing 
beliefs- and values on relationships between peo- 

I), Examines 6%ti beliefs and values and the inter- 
relationships between and among beliefs, values 
and behavior. . 

* 

Develops the competencies to acquire, organize 
and evaluate information for purposes of solving 
'problems and clarifying issues. 

A. Identifies problems or issues 'appropriate for 
study. . - 

B. Prepares a>plan to guide study of a problem or 
'issue. ' " • ^ 

^C. Identifies, locates and uses sources, of informa- 
tion and evaluates thq reliability and relevance 
of these sources. • • 

I). Organizes, analyzes, Interprets and synthesizes 
information obtained from various sources. 

E. Uses summarized information to draw conclu- 
.sions, offer solutions to problems or clarify is- 
sues. 

F. Validates outcome pf Study. ^ 

Develops the human relai^ion skillsonecessary to 
communicate ahd w(:|rk with others. 

A. Attends to expressjions of others. 

B. Encourages othera to express views and opin- 
ions. 

C. Listens carefully to others. 



n. Qlarifies and elaborates on own ideas. 

E. Asks for clarification and elaboration of the 

ideas of others. » 
f ' . Expresses awareness of difTerent discussion roles 

(e.g., initiator, facilitator, blocker) and 

recognizes some of the effects of these roles on 

individual and ^'roup action. 

G. Interacts in various capacides (e.g., leader, ad- 
* visor, supporter). 

H. Expresses willingness to interact with a variety 
of people. . 

I. Provides emotional and intellectual support for 

others in group efforts. 
J. Shares in, responsibilities that arise from group 

efforts. ' • 

V. Develops a positive si*lf-con<cept, jbuilds self- 
esteem and moves toward self-actualization. 

A. Expresses awarepess of the characteristics that 
qivc one identity^ ^ 

B. Expresses awareness of one's goals (aspira- 
tions), the goals of the groups with which one 
identifies and the V\\ between these goals^ 

'f^/ ' gxpr;j^|^.^^ of the relative strengths of 

^ oneself and the groups ^vith which one identifies 
and recognizes the societal barriers to full 
development that may exist. i 

1). Assesses the extent to which one has control 
over the setting and achievement of personal 
goals in light of what one knows about oneself, 
the groups with which cnc identifies and the 
societal barriers to full development. • 

E. Suggests ways to maximizing one's effectiveness^. 

VI. Develops and demonstrates a commitment to the 
ri|ht of self-determination for all human being* 
%xiA a willingness to take rational action in sup-- 
port of means for securing and preserving human 
rights. - ,^ 

A. Displays an awareness of a quality of human life 
and an interest in^ ways in which the quality can 

' be improved. 

B. Explains and supports rights and freedoms im- 
portant to human developmei^t. 

C. Participates in family, school and ^community 
life on the basis of rational decisioijis involving 
one's own values and the conflict among \\it^^ 
values. 0 ' 



ia-YEAR-OLDS . 

cl. Develops a kno\tledg6 base fgr understanding 
the rejlxtionships .between tiuman beings and 
^ their social and physical environment. 

A. Acquires knowledge abou' social organization. 

B. .Acquires knowledge about the relationships 
betv^een human beings and their social environ- 
ments and understands some of the conse- 
quences of these relationsl^ips. 
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Acquires knowledge about /he reiStionships 
between human beings ancr their physical en- 
vironments ^nd understands spme,oT the conse- 
quences ot thcfse relationships. . 
"Acquires knowledge about^ . decision-making 
processes. • \ 

Acquires knowledge about conflict and the im- 
pact ^t has on individual and group 
rclatibnshipL. 



III. 



Develops an understanding of the origini and in'- 
terre!ationshipt of beliefs^ values and behavior 
patterns V 

A. Expresses awareness of 5ome of the beliefs and 
values expressed by peop^le and recognizes that 
the conditions, times and places in vvhich people 
live influence t.ieir beliefs^ values and behaviors. 
Understands ways beliefs and values are trans- 
mitted in various cultufe^. 
Understands some of the influences of differing 
beliefs and values on relationships between peo- 
ple. 

Examines own beliefs and values and the inter- 
relationships between and an^ong beliefs, values 
and behavior. 



n. 



Develops the competencies to acquire, organize 
the evalu ate info rmation for purpose of solving 
problems and clarifying irsues. 

A. Identifies problems or'issues appropriate for in- 
vestigation. 

B. Plans how to investigate a problem or issue. 
C Identifies, locates and uses sources cf informa- 
tion and evaluates the reliability and relevance 

* of these sources. 
Organizes, analyzes, interprets and synthesizes 
information Obtained from various sources. 
Usei summarised information to draw conclu- 
sions, offer solutions to' problems, clarify issuer 
or make predictions. 
Validates outcomes of investigation, 
.\ppraise5 judgments and values that are in-^ 
volvecl in the choice of a course of actidp*^^ 



I) 



E. 



F. 
G. 



IV. Develops the human relation^itls^ necessary to 
communicate and w«orkj(vifh others. 

A. • Attends to expressions of others. 

B. Encour^es" others to express views and opin- 

Ignsr. 

C Listens carefully to others. 

D. Clarifies and elaborates on own ideas. 

E. Asks for clarification and elaboration of the 
ideas of others, 

, F. Expresses awareness of different discussion roles 
(e.g., initiator, facilitator, blocker) and 
recognizes some of the .effects of these roles on 
individual and group action. 

Ci. Interacts in various capacities (e.^., leader, ad- 
visor, supporter). • 

H. Expresses willingness to interact with a variety 
of people. 

I. Provides emotional and intellectual support for 

others in group efforts. 
J. Shares in responsibilities that arise from group 

efforts. 



V. Develops a positive self-^concept, builds self*^ 
esteem and moves toward ^selfrActualization.» 

' A. Expresses awareness of the characteristics that 
giVfe*one identity. 

B. Expresses awareness of one*s goals ;(aspira- 
tions), the goals of the groups with which one 
identifies and the fit between tl\ese goals. 

C. Expresses awapjncss of the relative strengths of 
oneself and 'the groups with which one identifies 
and recognizes the societal barriers to .full 

• dewlof)ment that may exist, 

D. Assesses the extent to which one has control 
over the setting . and achievement of personal 
goals in light of what one knows atuiut oneself, 
the groups with whicI|>one identifies and the 
societal barriers to fCiU dfevelopment, 

E. Suggests ways of maximizing one*s effectiveness. 

VI. Develops and demonstrates a commitment to ^he 
right ot self-determination for all human' beings 
and a willingness to take rational action in sup- 
port of means for securing and preserving human 
, rights. • - 

A. Dispi^ys an awareness of a quality of human life 
and an interest in ways in which the quality can 
be improved. 

B. Explains and supports rights and Freedoms im- 
portant to human development. " ^ 

C. Participates in family, school and community 
life on the basis of rational decisions involving 
one's own valucs^nd the conflict among these . 
values. . " . ^ 



•17-YEAR-OLDS AND ADULTS 



I. Develops a--kfibwledge base for understanding 
th^Teiationship between human beings an^their 
-'"^cial and physical environment. 

A. Acquires knowledge about social organization. 

B. Acquires knowledge about the relationships 
between human beings and their social 6nviroi» 
ments and . understands some of the conse- 
quences of these relationships. ^ 

C. Acquires knowledge about the relationships 
between human beings "and their physical ^n- 
■^wponments and understands st)^e of the conse- 
quences of these'rtlationships. . 

1). Acquires knowledge about decision-making 

processes. • ' - » 

E. Acquires knowleide about conflict and the im- 
pact it ' has on individual and group 
relationships.'' 

II. Develops an understanding of the origins tind in- 
terrelationships beliefsy^valifeS' and behavior 
patterns. • 

A. Expresses awareness of {5<5me of the tpeliefs and 
values expressed by people and recognizes that 
the conditions^ times and places in whi^h people 
live influence their beliefs, values and behaviors. 

B. Understands ways beliefs and values are trans- 
mitted in various cultures. 
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C. Understands* ibme oT the i^ifluiCiires of differing 
beliefe and v alues on relationships between peo- 

D. Examines own Reliefs and values and thr inter- 
/r relationships between ahd among beliefs^values 

andj behavior. 

^ ' 

Develons the comi^etencies to acquire, orgam|;c 
and evaluate information for purposes bff solving^ 
problems and clarifying issues. 

'A. Idenlfifies problems or issues appropriate for in- 
vestigation. . ^ 
» B. Designs a plan to investigate a problem or issue. 
C. ;crldctttiTies, -locates Ih^ uses spurces of informa- 
tion and evaluates the reliab'Bily and relevancy 
of these sources, "^i ^ 
I). Organizes, analyzesj^terprets and synthesizes 
*^ information 'obtained from various sources. 

E. Uses summarized information to draw concla- 
o sionsf offer solutions to problems, clarify issued, 

make predictions, or serve as a guide to con- 
tinued^ investigation. 

F^- Validatc$ outcomes of investigation. 

G. ' .Appraises judgments and" values that are in- 
volved in the choice of a course of action. 

Pevelops the human relation skills necessary to 
communicate and work with others. 

A. Attends to expre.^Jsions/ of others^ ^ 

B. Encourages others to express views ahd 
opinions. . ' P 

C. Listens carefully to others. 

>). Clarifies and elabormcTBtTMi^n ideas. 

E. Asks for clarification and elaBofatteajjf the 
ideas of others. 

F. Expresses awareness of different discussion roles 
(e.g., initiator, facilitator, blocker) ^nd 
recognizes some of the efforifs 9f these roles on 
idividual and group action, j 



G. Interacts *n various capacities" (e.g., leader, ad- 
visor, supporter). ' 

H. Expresses willingness to interact with a variety 
, of people. • - / 

I. Provides emotional and intellectual support for 

,pthcrs in group efforts. 
J. Shares*- in responsibilities that arise from group 

''fforts. 

V. ^Develops a positive self-concept, builds self- 
esteem and moves toward' self-actualization. 

A. Expresses awareness of the characteristics that 
give one identity. 

Expresses awareness of one's goal§ (aspira- 
trons), the goals of the groups with which one 
identifies and the fit between these goals. 
Expresses awareness of the relative strengths of 
oneself and the groups with which one identifies 
and . recognizes the societal barriers to full 
4 development. ^ / • 

Assesses the extent to waich one has control 
over the setting and aq^hievement of personal 
goals in light of what one knows about oneself, 
the groups with which one identifies and the 
societal barriers to -full development. 
Suggests ways of maximizing one's effectiveness. 



B 



C. 



D. 



E. 



VI. Develops and demonstrates a commitment to the 
right of self-detejrmination for all l^uman beings 
. and » willingness to take rational action in' sup- 
port of means for securing and preserving human 
V rights. , 0 

A. Displays JUi^warcness pf a quality of human lif< 
an interest in ways in which the quality can 
be imporved. 

Explains and supports rights and freedoms im- 
portant to human development. 
Participates in family, school and community 
jffe on (he basis of rational decisions involving 
oneV<iivn values and the conflict among these 
values. 



B. 



C 
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APPENDIX C 



: EVALUATION OF PERFORMANCE 
ON SELECTED SOCIAL STUDIES EXERCISER 



This appendix covers only a portion of the released exer- ' 
cises and is not intended to be a representative sample. A 
complete report on all rdeased exercises may be found in 
The First Social Studies Assessments An Overview, listed in Ap- 
pendix D. The panel's task (see Chapter 7) was not to 
predict what preformartce was to be but to judge wHat per- 
formance (realisticaUy) ought to be. The panel's judgments 
are no more, but no less, than those of nine competent social 
studies educators. Readers will do well to ask themselves 
with wh^t performance-^ levels they arc satisfied. 

This appendix is dividecl fnto two pirl;s. The first provides 
examples of exercises covering knowledge, skills and at- 
titudes in {general. The second section lists all the released 



exercises used by the Cox committee (see Chapter^. The 
Cox committee also reviewed 22 unreleased exercises. 

/ 



Key to exercise numbering system: 1st letter — R: released; 
2nd letter — S: skills, A: attitydes,^: knowledge; 3rd letter 
— O: obtaining information, I: interpreting information, R: 
rights of the lit^mcnidmentrftrworth'of the individual, E: 
economics, G: geography, H: history, P: political science j 
4th and 5th numbers are the numbers of the exercise within 
that category; 6th-9th letters would be subquestions within a 
multiple-part exercise. ^ 
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Sample Exercises from the 
First Social Studies Assessment 



RSt2QeD • ^ -/^ 

The Amcricah Declaration of Independence stat^: ' 

'*We hbld these truths to be self-evident, that all men are 
created equal, that they arc endowed by their Creator with cer- 
tain unalienable Rights, that among these arc Life/Libertyi and 
the pursuit of Happiness — That to secure these rights, Govern- 
ments ai'e instituted among Men, deriving their jufst powers from 
the consent of the gbverned ..." / * 

Do the following sentences restate ideas expressed in this quota- 
tion from the Declaration of Independence? 

C. Some rights can never*be rightfully taken away. 
Yes 

^ No . • • 

I don't know 

p. Governments get their right to govern people from the 
people. 

Yea 

No _■ 

I don't know 



Nat*l/Lcvcl of Acceptable 
Tfu-formancc; Actual 
/ (Percent) 



Realistically S^^tisfactory 
Pcifcrmance Level: Panel 
(Percent) 



Age Age Age Adult Age Age Age Adult 
9 13 17 ' 9 13 17 



— 75 



80 



80 76 ^ 61-80 >80 >80 



90 94 



61-80 • >80 >80 



RKEI6 / 

7r"7\nnajor American manufacturing corportat ion seeks to establish 
a branch plant in a country that has rich natural resources -but 
very little industry. The leaders of the nation turn down the 
American corporation's request. 

What reasons can you give for the decision made by the leaders 
of the foreign nation? 

RKHI3ABCD 

American Indians, Black Americans, Oriental Americans and 
Spanish-speaking Americans have contributed a great deal to the 
history and culture of our nation. For each of the groups i read 
« tell me the names of as many famous or nationally known men 
and worricn as you can. The person named may be either living 
or dead. Briefly describe each person's' contribution or jpeld of 
work. Consider people in ANY field of work — the l^Arts, 
Business^ Civil Rights, Education, Entertainment, Politics, 
Science or Sports. 

A: American Indians 

B. Black Americans 

C. Oriental Americans ^ / 

D. Spanish-speaking Americans 

RKH18 

Which one of the following is the MAJOR goal of the United Na- 



tions? 



To fight disease 
To maintain peace 
To spread democracy 
To fight the Communists 
I don't know 



47 



52 



52 — 



61-80 61-80 



18 
34 

0 

3. 



77 



35 

64; 
1 

7 



92 



41 6K80 61^80 61-80 61-80 

73 61-80 >80 >80 >80 

4 41-60 61-80 .61^80 61-80 

20 61-80 61-.80 61-80 61-80 



89 41-60 61-80 61-80 61-80 
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Sample Exercises from the 
First Social Studies Assessment 



Nat*l Level of Acceptable 
Performance:* Actual 
(Percent) 



Realtritically Satisfactory 
Performance LeveU Panel 
(Percent) 



RKE04 

Billy* went to a department store with his mother. As they went, 
into the store, Billy saw a sign in the store windbw. The sign 
looked like this: 



Age Age Age Adult Age Age Age Adult 

9 , 13 17 9 13 17 



BUY NOW 

PAY LATER 



Which one of the following t^ik vyhat the sigij means? 
The store is having a sale. 
Sbiric things in the store are free, ^ 
You can pay at another tinie for what you buy today. 
You can trade something you have for something you 
want. . * 
I don't know. „ 



89 



- „ - 6US0 ^ 



RKE09BC 

Which of the following things happen when a country becomes 
highly industrialized? 

B. There is |;reater emphasis on individual craftsmanship, 
r don't know 



G. There is a movement 6f people from rural to urban pans of 
the country. ^ 
Yes • 

; No • ■ 

i I don't know 



41 



— 67 



66 '53 



78 



83 — 



41-60 61-80 61-80 



61-^0 61-80 61-80 



RkEOt , , . 

Economists divide purchases into two groups: producer goods 
and consumer goods. A farmer buys seed, fertilizer, a tractor and 
a new coat for his wife. . 

Which one of the thingi that the farmer bought is in the group of. 
consumer goods? 
Seed 

Tractor \ , 

Fertilizer 

New coat for his wife 
I don^t know 



— 57 



81 



79 



61-80' 61-80 >80 



RKG05 

Which one of the following states borders on the Atlantic Ocean? 37 
California " 
Nebraska • 
New York 
Ohio 

! don't know 



61-80 — — 
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Sample Exercises from the 
First Social Studies Assessment 



RKEl'2 

The term "monopoly" describes the situation in which the 
market price of goods snd services is established by which one of 
the following? . i 

Many sellers . 

ArSingle buyer 

Many buyers and sellers 

A single seller or a small group of sellers 

I don*t know 



NatM Level of Acceptable Realistically Satisfactory-*^ 
Performance: Actual Performance Level: Panel 
(Percent) (Percent) 



Age 

9 . 



Age 
13 



Age Adult. Age Age Age Adult 



17 



21 



13 



17 



30 ^ — 61-80 61--80 



RSI31 

Look at fhe cartoon. What idea is the artist trying to put across 
in this cartoon? 



--^ 83^ 88 



91 _ 61-80 



^80 >80 




Cartoon^ by Herbert Block, ^'Richest 
Country in the World," from Herbhck's 
line and ,Yow (New York: Simon & 
Schuster/ 1955), p. 273, Reprinted by 
permission of Herbert Block. 



^ • RS007ABCDE 

A. What do you think are three innportant problems facing large 

cities in the United States? 38 59 66 — 41-60 61-80 >80 

B. Which one of the problems you named would you MOST 

f want to ask questions about?* r ^ ' ■ * 

C: What* two questions would you ask about this problem to fmd 
out jnore about it? _ 

D. Name two sources that would help you learn more about the 

problem you most want to ask questions about. / ^ . » 

E. Name as many additional sources as you can that would help 
you learn more about the problem you MOST want to ask 
questions about. 
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Sample Exercises from the 
First Social Studies Assessment 



Nat'l Level of Acceptable 
PerfornSiigs^ Actual 
(Percent) 



lt,ealiitically S^itiifactory > 
Performance Level: Panel 
(Percent) 



Age* Age Agt Adult Age Age Age Adult 
9 13 17 ^9 1^ . 17 



RS0t4CD . 

Suppo&e someyane wanted to know what had "happened in our 
country or in the world within the last 24 hours. Would each of 
th£ following be a good way for that person to find out? 

C. Would reading the daily newspaper^be a good way to find out 
what had happened in, the 'last 24 hours? 

D. Would looking in an encyclopedia be a good way tq find out 
what had happened in the last 24 hours? 



87 ' 98 — — 



89 98- 



>80 >80 
>80 >80 



RSI23 



Look at the graph; then answcrthe question below it. 



iNtHlYp 



MONtHLY AVERAGE RETAIL PRICES 
FOR EGOMI(NI> APPLES IN CHICAGO 

(ifsd^ma) 




40 




Dec 



Reprinted by permi^ion of the Bureau of Labor 3tatistics 



The average retail price of apples was highest in which month? 
Jjanuary . ^ 

July , / 

■ October 
December 

I don't know - ^ 



— 89 96 91 



61-80 >80 >80 
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Sample Exercises from the 
First Social Studies Assessment 



NatM Lcverof Acceptable 
0 Performance: Actual 
{percent) 

Age Age Age Adult 
9 13 17 



Realistically Sattifactotry 
Performance Level: Panel 

-tPcrcent) & 

kge Age Age Adult 
9 13 17 



RSI33 . 

Please listen carefully to this recording of "Carefully Taught'* 
from South Pacific- "VVhile listening, try to identify thfJ main idea or 
message. • 



You Ve got to be taught 

To hati^ and fear . . » 
*^ . You*jt got to be taughlt ^ 

From year to year • * 

It's got to be drummed * 

In your flcar litftle 

YouVe got t,o be careiully taught 

YouVc got to be taught 
' * To be afraid 

' ' ^' , ' - " Of people whose eyes ■aj'e oddly made • 
y \ o And people whose skin i^ a different shade 

YouVe got to be carefully taucdit., 

YouVe got to be taught befor? it's too late 

Before you arc 6 or 7 or 8 

To hate all the [)copIe ,^ 

Your relatives hate. • 

You/ve g6t carefully taught 

YouVe got to be carefully taught. 

What is the song ^soutX^ . 

RSOll / \ 

A boy looked in his history bobki but he could not find out where 
Abraham Lincoln was born. Which one of the following should 
he do? ' \ . 

Look in an atlas ^ 

Look in an encyclopedia 

Look in a geogr^hy . ' 

Ask a friend to help him 
1 ,1 don't know 

RS117C - . . 

^ Some things CAN be proved to be true or false, some things 
CANNOT BE proved to be true or false. v 

^ Read each of thp statements below and decide whether it c^n or 
cannot be proved. If you think it CAN be proved, fill in the oval^ 
beside "Can be proved. "If you think it CANNOT be proved, fill ' 
in the oval beside **Cannot be proved." If you do not know the 
answer, fill in the' oval beside **I don't know." An example is 
* doiie you. 

Example * . 

ITie earth is almost round. / ' 

Can btf* proved ^ ^ 
* ' ^ Cannot be proved 

1 don't know 

^ . ■ ■■<> . • 

C. People from Mexico are nicer than people from Canada. 
Can be proved 
, Cannot be proved 
I don't know - ' , 



/ 



22 



39 



68 



39 — 61-80 ' >80 >80 



61-80 — 



53 



79 



72 61-80 61-80 >80 >80 
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Sample Exercisei^ from the 
First Social Studies Assessment 



Nat'l, Level of Acceptable 
\ , Peiformance: Actual 
. (Percent) 

^gt Age Age Aduit 

9 - 13 %7 



RABll 



Stippose a friend from India comes to yout^ house for dipner. 
Vour mother is making hamburgers for diiyxer/ While ypu arc 
playing, your friend tells you he docs not eat mc^^ His fdigio^i , 
wtU 'hot let him eat meat. What should ^yqu do? > ' 



RAB04 



Do you think the peoplcVho live in a neighborhodd should be at- 
lowed'to decide who can and cannat Jive in their^neighborhood? — 



80 



RKP01 • ' ^i^r^-^ " r/ ' . . . . 

Below are'^listed four of the many jobs that arc don^ rn'a,iity. 
Which one of the jobs is done by the health* department? r, ' 
Selling food 

Dfrecting traffic - . i 

Putting out fires * ; . 

Inspecting restaurant^ ** ^ . • . * , 

1 don*t know * 



36 



Realistically Satisfactory^ 
Pefforn^nre Level: Panel 
^ (Percent) 

Age Age Age Adtilt 
9 13 17 



63' 



61-80 — 



4-: 



— >80 >80 



^ — 61-80 — ^ 



. f 



RKP02 . - ' V ; 

In a court, which one of the following ha^ the job of making sure ' 
that 'the trial is fair and run according to the ruies?- ? ' ^ 
The judge ^ ^ vV;-^ 

The lawyer . * '..,4^ 

The jury ^ ' 

■ ThC! person on trial *^ • - • . ^ 

I don*t knbw , . • 

RKP16 . \ 

In the United States which one of the following men is elected to 
office? , ^ ; 

^ A United States senator ^ . : , 

The United States secretary of state 

A United States Supreme Court justice 

Thfc United States ambassador to Qreat Britain;^ ^ 

I ddh't know* 



74 . — 



61-80 — ^ ^ 



89 



901 



— 61-^80 61-80 >m 



RKF17 . . ' 

' - - ■ ■ ^ ^. , 

T4ie presidential candidate f6r each. major political party is' foN 
mally nominated by which one of the followir^? ^. ' 

The Senate " . / ' ' 

A national primary " ^ ' 

A national convention ■ , 

The House of Representatives '* ^ * * ' r ' ' 

I don*t know " ■ 



1.7 49 



60 

''2\ 



61-80 61-80 61-60 



RKP13 

Which one ef the following has the power to decfare an a^fdf 
Congress unconstitutional? , * 

, The Congress . \. i . 

The president ^ 

«3Tie United States Supreme Court ^ 
The.Unhed States Department of'jus^tice ^ ^ 
I don *t' know 'i' 



35 



71 



62 — 61-80 80 61-80 



c • 



"' "Slml^le^ Exercises froth the " 
First Social Studies Assessment 



\ 



Nat'l. Level of Acceptable ' Realistically Satisfactory 

Performance: Actual Performance Level: Panel 

^Percent) (Percent).' 

Age I Age Age Adult Age Age Agei Adult 



13 17 



.13 



17 



0) 



RKP03 ' \ • ■ • 

The head of governme^nt in the United States is the president. 
.Which one of the following is usually the head of government In a 
town? . ^ 

', The niayor \ ^ ' >^ 

The governor . A 

The chief of police \ * ^ . 

The school principal " ; 

I don*t knov/1 ,\ : » 



58 ^ 



_ 41.60 ^ ~ — 



RKP18 



used 



The. ballot below was used in a^eneral election. Look at ballqfcto answer the questions on this and the following Iwo pages. 



) 

OFFICES 


y- LEGISLATIVE 


* COUNTY 


^ ^SENATOR 
' IN.qONGRESS 
^(vote/or one) 


REPRESENTATIVE 
IN CONGRESS 
(vote for one) 


^ COUNCILMAN 
(vote for two) 


TAX ASSESSOR 
(vote for^ne) 


DEMOCRATIC 


AlanF' 

KIRK 

• * ' ~ — - 


JohnG 
SMITH ^ 


Martha G 
DAVIS 


. PrterV 
MOSS 




REPUBLICAN ■ 


' , 'James' M 
" ' JONES^ - 


Mary 
O^CONNOR 


John 
RICHARDS 


Michael M 
MERWTN 


'Joseph L 
LASKr 



A. If you wanted to vote for Kirk foir scnajor, could you also vote 
for 0*Connor for ihember ofthe House. 'of Representatives? 

Yes/ . 

Nov ■ 

.1 don't kiiow j , ^ 

B. Could youi vote for .4?i5«h Davis and Moss for councilman? 

C. Could you vote for b^th Davis and Merwii^ for councilman? 

D. If you were regiifered ^s a meniber^ofthe Democratic Party, 
could you vote fot Las'ki for tax assessor? ' 

E» Could you vote, for lx»th*Kirk and Jones senator? 

" .-. "* ■ ' , 

RKP08B ^k-V .^ ^ " ^ 

Which one of the foltowing \^ould MOS^ likely pass an act to 
raise the rajles for sepciing hctters thrdugli the mail? 

Federal goyerftmenf^^'. 

State government , ' ■ • 

Local government ^ " • . 
• <r I don't know ' ^ ' ' V 

. ^ - • ^; .■ ■ . ' 

RKP08C . ^ ^ " 

Which one of the following* woxild MOST likely pass an act to 
lower taxes on goods comingfinto the coutitry? 

Federal government • 

Slate governilynt *' 

Local governnient .'n ^ 

I don't know ' ' r. * ' 



— 41 44 



^ 72 90 



73 89 



92 



— >8(f >80 

O 



95 61-^0 01-80 >80 



-r- 61-80 >80. >80 
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Sample Exercises from the 
First SocialiStudies Assessment 



RKP08D 

Which one oftKc following would MO&T likely pksi an act to in- 



crease garbage collection services? 
Federalgbdcrnment 
State goveriiment 
Local ^ovefnment 
i don't krt( 



RKPll 



The Supreme Court ruled that it is unconstitutional to require 
prayer and forma religious instruction in public schools. Which 
one of the fdllowl ig was^^fe basis for its decision? 

The requiitments violated the right to freedom of speech. 
Tl^erc wasrstrong pressure put on the Supreme Court by 
tertain religious minorities. °, 
Religious exercises violated the principles of the separa- 
tion of church and state. 

Every moment of the valuable school time was needed to 
^^repare students to earn a living. ^ 
I don^t know. 



NsU'l Level of Acceptable 
Performance: Actual 
(Percent) v^^> 

Age , Age Age Adult 

9 13 ' 17 



Realistioiily Satisfactory 
Performance Level: Panel 
(Percent) 

.Age Age Age Adult 
9 13 17 



77 



92 . 92 ^ 61-80 >80 >80 



49 



52 



— . >8P 61-80 



RAB17 ^ " 

» 

Should a congressfnan pay attention* to the opinions and con- 
cerns of people whose views are different from those of the ma- 
< jority? 

Yes 
.No 

I don't know 



Please explain any answer you^ selected.. 



79 



78 . — 



•61-80 61-80 



RAR05 " > . 

Indicate whether you agree or disagree with the following state- 
ment: Anyone vvho criticizes , the use of United States troops in 
military action ^abroad should be prohibited from expressing his 
views publicly. ' 

Agree ^ ^ 

Disagree ^ 

Please explain your position. 



— — 81 61 



>80' >80 



RARll ^ 

In the picture, there are many people gathered togethel- in a 
public park. They are demanding changes which you do not 
agree with. Should these people be allowed to gather and make 
their demands in a public place? 
Ves 

Nt) . . , 

Undecided ^ 
No response . 



54 80 72 ' — " 61-80 >80 >80 



113 



ERIC 



Q0119 



* ' ^Sample Exercises from the > Nat'l Level of Acceptable Realistically Satisfactory 

First Social Studies Assessment ' Performance: Actual Performance Level: Panel 

^ * (Percent) (Percent) 

i ' Age Age ^ Age Adult Age Age Age Adult 

9 13 17 9 13 17 0 

RAR08 ^ 

Shoiiid a newspaper or magazine be allowed to publish , 

something that crhicizes an elected government official? . — 41 66 71 — ->80 >80 >80 
Yes 

No. . . ' ' ■ ^ ' ^ : 

Undecided ' , ' 

RAR07 

Should a person who does not believe in God be allowed to hold a 

public office? 59 63 56 — 61-80 >80 >80 

Yes * . 
No / * 

Undecided . - • 

RAR12A . 

Do you think people should be allow/sd to picket the holding of a ^ * 

rock festival as a protest against it? — -j- 4? — * — >80 >80 

Yes ' ^ 
No ; ^ . \ , 

Undecided 

<?* 

RAR12B ' ^ * ^ - 

Do you think people should be allowedi to picket a police station 

to protest reported police brutality? « — — ■ 40 44 — — >80 >80 

Ves ■ . 

No * 
Undecided 

' RAB2d" • ^ 

Do you think there should be laws against acts of varidalism such • ' 

as destEQying a statue? , • — 55 51 — — >80 >80 — 

tYcs. ■ V . 

^ • No ^ , ^ .V 

Undecided 

RAB16 ^ 

Should race be a factor in hiring someone for a job? » 74 90 90 — >80 >80 >80 

Yes " , . ' 

No ♦ , • \ 

I don't know 

* ♦ 
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